Nov 01 08:29:29.689501 ip-10-0-143-16 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 08:29:29.689518 ip-10-0-143-16 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 08:29:29.689528 ip-10-0-143-16 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 08:29:29.689892 ip-10-0-143-16 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 08:29:39.722462 ip-10-0-143-16 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 08:29:39.722484 ip-10-0-143-16 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 56d7ba65b81a402b94495371b404cc68 -- Nov 01 08:31:42.801066 ip-10-0-143-16 systemd[1]: Starting Kubernetes Kubelet... Nov 01 08:31:43.173304 ip-10-0-143-16 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 08:31:43.173304 ip-10-0-143-16 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 08:31:43.173304 ip-10-0-143-16 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 08:31:43.173304 ip-10-0-143-16 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181402 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181514 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181519 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181523 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181526 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181529 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181532 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181534 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.181521 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181537 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181540 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181543 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181546 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181549 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181551 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181554 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181557 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181559 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181562 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181565 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181568 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181570 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181573 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181575 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181578 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181580 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181583 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181586 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.181876 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181588 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181591 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181593 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181596 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181604 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181607 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181610 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181612 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181615 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181617 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181620 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181623 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181625 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181628 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181631 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181633 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181637 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181639 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181642 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.182323 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181644 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181647 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181649 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181652 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181656 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181658 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181661 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181664 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181666 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181668 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181671 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181673 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181676 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181678 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181681 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181683 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181685 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181688 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181690 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181693 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.182802 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181696 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181698 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181701 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181704 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181706 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181712 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181730 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181733 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181736 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181739 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181743 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181748 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181751 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181754 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181756 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181759 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181763 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181767 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.183311 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181770 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181773 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181776 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181778 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181781 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181783 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181786 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181788 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181791 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181793 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181795 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181805 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181809 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181811 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181814 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181817 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181820 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181824 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181827 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.183790 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181829 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.181832 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182758 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182768 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182773 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182777 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182780 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182783 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182786 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182788 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182791 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182793 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182797 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182800 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182802 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182805 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182808 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182811 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182813 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.184236 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182816 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182819 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182822 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182827 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182829 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182831 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182834 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182837 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182841 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182843 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182846 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182850 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182853 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182856 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182858 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182861 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182863 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182866 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182868 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.184707 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182871 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182873 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182876 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182878 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182881 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182884 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182888 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182892 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182897 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182901 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182904 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182906 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182909 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182912 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182915 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182917 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182919 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182922 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.185201 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182924 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182927 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182930 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182933 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182936 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182938 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182941 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182943 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182946 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182948 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182951 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182954 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182956 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182958 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182961 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182963 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182966 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182968 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182971 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182973 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.185810 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182975 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182978 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182981 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182985 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182989 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182992 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182996 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.182999 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183003 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183005 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183008 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183011 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183013 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183016 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183019 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183021 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183024 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183027 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183029 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183032 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.186285 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183034 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183036 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183039 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183041 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183044 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183047 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183049 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183052 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183054 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183057 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183059 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183594 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183599 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183603 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183606 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183608 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183611 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183614 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183616 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183619 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.186787 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183622 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183624 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183627 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183629 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183632 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183635 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183638 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183641 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183643 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183646 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183648 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183651 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183653 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183656 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183658 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183661 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183664 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183666 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183669 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183672 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.187260 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183674 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183680 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183682 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183685 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183687 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183690 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183693 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183696 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183699 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183701 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183704 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183707 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183709 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183711 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183729 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183732 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183736 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183740 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183743 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.187758 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183746 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183749 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183752 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183754 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183757 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183764 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183767 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183770 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183772 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183775 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183777 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183780 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183783 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183785 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183789 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183792 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183795 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183798 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183801 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183804 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.188203 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183807 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183810 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183812 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183815 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183817 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183820 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183822 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183825 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183829 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183833 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183836 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183839 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183842 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183844 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183847 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183849 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183852 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183854 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183857 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.188664 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183860 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183862 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183865 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183867 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183870 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183872 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183875 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183879 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183882 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183885 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183887 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183890 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183892 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183895 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183898 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183900 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183904 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183907 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.183909 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.184974 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 08:31:43.189128 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.184991 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.184998 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185003 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185008 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185011 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185016 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185020 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185024 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185027 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185030 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185035 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185039 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185044 2580 flags.go:64] FLAG: --cgroup-root="" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185047 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185050 2580 flags.go:64] FLAG: --client-ca-file="" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185053 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185056 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185062 2580 flags.go:64] FLAG: --cluster-domain="" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185066 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185070 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185073 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185077 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185081 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 08:31:43.189592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185084 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185087 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185090 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185093 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185097 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185100 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185104 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185107 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185111 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185114 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185117 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185120 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185123 2580 flags.go:64] FLAG: --enable-server="true" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185126 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185131 2580 flags.go:64] FLAG: --event-burst="100" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185134 2580 flags.go:64] FLAG: --event-qps="50" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185137 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185140 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185143 2580 flags.go:64] FLAG: --eviction-hard="" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185147 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185150 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185155 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185158 2580 flags.go:64] FLAG: --eviction-soft="" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185161 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185164 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 08:31:43.190156 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185167 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185170 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185174 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185177 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185180 2580 flags.go:64] FLAG: --feature-gates="" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185185 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185188 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185191 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185195 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185198 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185201 2580 flags.go:64] FLAG: --help="false" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185205 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185209 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185212 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185215 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185219 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185222 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185225 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185228 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185231 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185234 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185237 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185241 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185244 2580 flags.go:64] FLAG: --kube-reserved="" Nov 01 08:31:43.190785 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185247 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185250 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185253 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185255 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185259 2580 flags.go:64] FLAG: --lock-file="" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185262 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185265 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185268 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185275 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185278 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185281 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185284 2580 flags.go:64] FLAG: --logging-format="text" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185287 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185290 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185293 2580 flags.go:64] FLAG: --manifest-url="" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185296 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185301 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185304 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185308 2580 flags.go:64] FLAG: --max-pods="110" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185311 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185314 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185318 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185321 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185324 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185327 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 08:31:43.191350 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185330 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185339 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185342 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185345 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185348 2580 flags.go:64] FLAG: --pod-cidr="" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185351 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185354 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185357 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185360 2580 flags.go:64] FLAG: --port="10250" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185363 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185366 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0506531ba9bd3669b" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185370 2580 flags.go:64] FLAG: --qos-reserved="" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185372 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185375 2580 flags.go:64] FLAG: --register-node="true" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185378 2580 flags.go:64] FLAG: --register-with-taints="" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185383 2580 flags.go:64] FLAG: --registry-burst="10" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185385 2580 flags.go:64] FLAG: --registry-qps="5" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185389 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185392 2580 flags.go:64] FLAG: --reserved-memory="" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185396 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185399 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185402 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185405 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185407 2580 flags.go:64] FLAG: --runonce="false" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185410 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185413 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185416 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 01 08:31:43.191957 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185419 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185422 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185426 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185429 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185432 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185435 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185438 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185441 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185444 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185447 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185450 2580 flags.go:64] FLAG: --system-cgroups="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185453 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185459 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185461 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185464 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185469 2580 flags.go:64] FLAG: --tls-min-version="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185473 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185476 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185479 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185483 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185486 2580 flags.go:64] FLAG: --v="2" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185491 2580 flags.go:64] FLAG: --version="false" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185496 2580 flags.go:64] FLAG: --vmodule="" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185500 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.185503 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 08:31:43.192892 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185668 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185672 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185675 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185678 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185680 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185683 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185686 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185688 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185691 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185694 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185697 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185700 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185702 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185705 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185707 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185709 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185712 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185726 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185729 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185732 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.194005 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185734 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185737 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185739 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185744 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185746 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185751 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185755 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185758 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185761 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185764 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185769 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185772 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185774 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185777 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185779 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185782 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185784 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185788 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.194932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185791 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185793 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185796 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185798 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185802 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185804 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185807 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185810 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185813 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185815 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185818 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185821 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185823 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185826 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185829 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185831 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185835 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185837 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185840 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.195779 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185843 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185845 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185848 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185850 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185853 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185856 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185859 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185861 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185864 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185866 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185869 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185871 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185876 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185879 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185881 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185884 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185887 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185890 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185893 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.196680 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185895 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185898 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185900 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185903 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185905 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185908 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185910 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185913 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185915 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185918 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185921 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185924 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185926 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185930 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185934 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185937 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185940 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185942 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185945 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185948 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.197393 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185950 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185953 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185955 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185958 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185960 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185964 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185967 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185969 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.185972 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.186001 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186045 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186049 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186052 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186055 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186058 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186060 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.197945 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186063 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186066 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186068 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186071 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186073 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186076 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186078 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186082 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186085 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186088 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186090 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186093 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186095 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186098 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186100 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186103 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186105 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186108 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186111 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.198573 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186116 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186119 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186123 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186126 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186129 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186132 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186134 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186137 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186140 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186142 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186145 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186148 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186150 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186153 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186155 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186158 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186160 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186162 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186165 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186167 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.199377 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186172 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186174 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186177 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186180 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186182 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186185 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186187 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186190 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186192 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186195 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186197 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186200 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186202 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186205 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186209 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186212 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186214 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186217 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186220 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186222 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.200017 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186226 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186229 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186232 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186235 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186238 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186240 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186243 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186246 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186248 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186251 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186253 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186256 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186259 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186262 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186264 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186266 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186269 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186272 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186274 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186277 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.200639 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186280 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186282 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186285 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186288 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186290 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186293 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186297 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186299 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186302 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186305 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186307 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186310 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186312 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186315 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186317 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186320 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186323 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186326 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186329 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.201208 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186331 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.186334 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.194900 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.194931 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.194955 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.194962 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195059 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195068 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195073 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195078 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195082 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195087 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195092 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195097 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195102 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195106 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195110 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195115 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195119 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.201670 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195124 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195128 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195132 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195136 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195141 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195145 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195152 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195159 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195165 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195170 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195175 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195180 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195184 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195189 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195198 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195204 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195209 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195213 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.202207 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195217 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195222 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195226 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195230 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195234 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195239 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195243 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195247 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195251 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195256 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195260 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195265 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195269 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195274 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195279 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195283 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195287 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195292 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195296 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195301 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.202658 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195305 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195309 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195313 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195317 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195322 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195326 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195331 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195336 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195340 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195344 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195349 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195354 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195358 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195362 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195367 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195371 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195375 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195379 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195385 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.203371 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195392 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195398 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195403 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195407 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195411 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195415 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195420 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195424 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195429 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195433 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195438 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195443 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195448 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195452 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195456 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195461 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195465 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195469 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195473 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195477 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.204027 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195481 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195486 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195491 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195495 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195500 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195505 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195509 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195514 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195517 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195522 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195526 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195530 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195534 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195539 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195543 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.195591 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195862 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.204820 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195874 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195881 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195886 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195891 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195896 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195901 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195906 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195911 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195915 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195921 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195925 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195929 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195933 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195937 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195941 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195945 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195949 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195954 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195958 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.205555 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195964 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195971 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195975 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195979 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195984 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195990 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.195997 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196002 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196006 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196011 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196015 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196019 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196024 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196028 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196033 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196037 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196041 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196047 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196051 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.206363 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196056 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196060 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196064 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196068 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196072 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196076 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196081 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196085 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196089 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196093 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196098 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196102 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196106 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196110 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196115 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196121 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196125 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196130 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196134 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196139 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.206932 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196143 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196147 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196152 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196156 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196160 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196165 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196169 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196174 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196178 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196183 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196187 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196191 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196195 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196199 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196203 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196207 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196210 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196214 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196218 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196222 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.207426 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196226 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196230 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196234 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196239 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196243 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196248 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196253 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196258 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196264 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196268 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196273 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196278 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196282 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196286 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196291 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196295 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196299 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196303 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196308 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196313 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.208019 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196317 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196321 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196326 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196330 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196334 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196338 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196342 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.196391 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196444 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196450 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196454 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196459 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196463 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196468 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196472 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196476 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:31:43.208495 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196480 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196484 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196488 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196493 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196498 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196503 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196508 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196512 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196517 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196521 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196525 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196529 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196533 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196537 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196541 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196544 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196549 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196554 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196558 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196562 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:31:43.208997 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196566 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196570 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196574 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196578 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196583 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196587 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196593 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196599 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196604 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196610 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196614 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196620 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196625 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196630 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196634 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196638 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196643 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196647 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196652 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:31:43.209462 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196657 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196661 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196665 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196669 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196674 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196679 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196683 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196687 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196691 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196695 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196699 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196703 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196707 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196711 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196739 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196743 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196747 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196752 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196756 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196761 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:31:43.209982 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196765 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196769 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196774 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196778 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196782 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196786 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196791 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196796 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196800 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196803 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196808 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196813 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196823 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196827 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196832 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196836 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196840 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196844 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196849 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:31:43.210447 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196853 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196857 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196861 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196866 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196870 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196874 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196878 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196882 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196886 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196891 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196895 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196899 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196908 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196912 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196918 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196923 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196927 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196931 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:31:43.211030 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:43.196935 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.197035 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.200124 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.201049 2580 server.go:1014] "Starting client certificate rotation" Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.201152 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.201196 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.204097 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 08:31:43.211530 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.206261 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 08:31:43.217354 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.217330 2580 log.go:25] "Validated CRI v1 runtime API" Nov 01 08:31:43.221062 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.221041 2580 log.go:25] "Validated CRI v1 image API" Nov 01 08:31:43.221920 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.221893 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 08:31:43.226253 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.226227 2580 fs.go:135] Filesystem UUIDs: map[1ca88dd6-c6c4-4015-82fb-d3254545b3f7:/dev/nvme0n1p4 4fdad4ec-7f19-4642-afe3-aa58c6c757e7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 01 08:31:43.226332 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.226253 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 08:31:43.230815 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.230667 2580 manager.go:217] Machine: {Timestamp:2026-11-01 08:31:43.230178612 +0000 UTC m=+0.319684892 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3277442 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec259e04ad369ead7003a93869eeef76 SystemUUID:ec259e04-ad36-9ead-7003-a93869eeef76 BootID:56d7ba65-b81a-402b-9449-5371b404cc68 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ca:c6:08:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ca:c6:08:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:42:78:a1:51:ae Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 08:31:43.230815 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.230810 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 08:31:43.230943 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.230904 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 08:31:43.234289 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.234257 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 08:31:43.234460 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.234289 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-16.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 08:31:43.234552 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.234466 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 08:31:43.234552 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.234479 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 08:31:43.234552 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.234534 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 08:31:43.234658 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.234560 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 08:31:43.235565 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.235552 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 08:31:43.239932 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.239917 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 08:31:43.240091 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.240079 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 08:31:43.240765 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.240755 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 01 08:31:43.240836 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.240774 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 08:31:43.240836 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.240788 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 08:31:43.240836 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.240802 2580 kubelet.go:405] "Adding apiserver pod source" Nov 01 08:31:43.240836 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.240815 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 08:31:43.241218 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.241203 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 08:31:43.243441 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.243413 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 08:31:43.249301 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.249287 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 08:31:43.250279 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250266 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 08:31:43.250330 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250298 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 08:31:43.250479 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250468 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 08:31:43.250507 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250485 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 08:31:43.250507 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250492 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 08:31:43.250507 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250498 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 08:31:43.250507 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250504 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 08:31:43.250603 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.250511 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 08:31:43.251030 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.251021 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 08:31:43.251060 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.251031 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 08:31:43.251060 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.251044 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 08:31:43.251060 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.251051 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 08:31:43.251060 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.251060 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 08:31:43.251262 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.251068 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 08:31:43.252330 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.252319 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 08:31:43.252362 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.252333 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 08:31:43.256532 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.256518 2580 server.go:1263] "Started kubelet" Nov 01 08:31:43.256604 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.256560 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 08:31:43.256675 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.256558 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 08:31:43.256746 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.256730 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 08:31:43.257299 ip-10-0-143-16 systemd[1]: Started Kubernetes Kubelet. Nov 01 08:31:43.258044 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.257999 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 08:31:43.259432 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.259414 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 01 08:31:43.262549 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.262532 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 08:31:43.264591 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.264574 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 08:31:43.265698 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.265683 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 08:31:43.265896 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.265853 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:43.266742 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.266694 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 08:31:43.267913 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.267885 2580 factory.go:55] Registering systemd factory Nov 01 08:31:43.267913 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.267912 2580 factory.go:223] Registration of the systemd container factory successfully Nov 01 08:31:43.268139 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268125 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 08:31:43.268223 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268215 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 08:31:43.268290 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268273 2580 factory.go:153] Registering CRI-O factory Nov 01 08:31:43.268339 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268302 2580 factory.go:223] Registration of the crio container factory successfully Nov 01 08:31:43.268386 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268366 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 08:31:43.268431 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268386 2580 factory.go:103] Registering Raw factory Nov 01 08:31:43.268431 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268401 2580 manager.go:1199] Started watching for new ooms in manager Nov 01 08:31:43.268656 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268636 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 01 08:31:43.268656 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.268653 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 01 08:31:43.269835 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.269817 2580 manager.go:319] Starting recovery of all containers Nov 01 08:31:43.281826 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.281807 2580 manager.go:324] Recovery completed Nov 01 08:31:43.285834 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.285821 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.287592 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.287574 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.287639 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.287610 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.287639 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.287623 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.288191 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288178 2580 cpu_manager.go:226] "Starting" policy="none" Nov 01 08:31:43.288191 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288189 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 08:31:43.288281 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288207 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 08:31:43.288281 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288233 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.288958 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288937 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.289042 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288970 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.289042 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.288984 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.290425 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.290407 2580 policy_none.go:50] "Start" Nov 01 08:31:43.290425 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.290424 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 08:31:43.290564 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.290434 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 08:31:43.292349 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.292336 2580 policy_none.go:44] "Start" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.333444 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.337910 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.338111 2580 manager.go:341] "Starting Device Plugin manager" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.338160 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.338172 2580 server.go:87] "Starting device plugin registration server" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.338467 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.338479 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.339366 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.339462 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.339472 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.339662 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.339687 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.340150 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.340200 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.340381 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 08:31:43.347694 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.340436 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:43.395319 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.395274 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-16.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 08:31:43.395319 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.395293 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-16.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 08:31:43.396178 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.395275 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-16.us-west-2.compute.internal.18e3deab2478d6a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-16.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-16.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:43.256262308 +0000 UTC m=+0.345768587,LastTimestamp:2026-11-01 08:31:43.256262308 +0000 UTC m=+0.345768587,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-16.us-west-2.compute.internal,}" Nov 01 08:31:43.439041 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.438967 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.440068 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.440050 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.440161 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.440082 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.440161 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.440092 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.440161 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.440126 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.440302 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.440284 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal"] Nov 01 08:31:43.440361 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.440349 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.441169 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.441152 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.441265 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.441182 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.441265 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.441196 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.443543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.443527 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.443708 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.443693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.443784 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.443745 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.444393 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.444379 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.444470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.444405 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.444470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.444415 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.444470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.444379 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.444561 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.444486 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.444561 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.444502 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.446741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.446700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.446840 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.446763 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.447850 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.447827 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.447949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.447862 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.447949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.447883 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.462820 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.462793 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 08:31:43.462948 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.462879 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hnqk2" Nov 01 08:31:43.470006 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.469983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.470089 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.470015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.470089 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.470039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61735862d7ea6a76be3182acee32ca86-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"61735862d7ea6a76be3182acee32ca86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.483904 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.483882 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.486012 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.485994 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.492421 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.492393 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 08:31:43.492421 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.492394 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 08:31:43.492535 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.492410 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-16.us-west-2.compute.internal.18e3deab2656ed3b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-16.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-16.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-16.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:43.287594299 +0000 UTC m=+0.377100583,LastTimestamp:2026-11-01 08:31:43.287594299 +0000 UTC m=+0.377100583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-16.us-west-2.compute.internal,}" Nov 01 08:31:43.492596 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.492553 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 01 08:31:43.492696 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.492676 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-16.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 08:31:43.571041 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.571002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61735862d7ea6a76be3182acee32ca86-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"61735862d7ea6a76be3182acee32ca86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.571041 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.571038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.571285 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.571056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.571285 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.571116 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.571285 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.571120 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.571285 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.571121 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61735862d7ea6a76be3182acee32ca86-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"61735862d7ea6a76be3182acee32ca86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.588535 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.588503 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-16.us-west-2.compute.internal\" not found" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.695204 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.695110 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-16.us-west-2.compute.internal\" not found" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.789323 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.789288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.789496 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.789277 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:43.790482 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.790451 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:43.790482 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.790482 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:43.790639 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.790492 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:43.790639 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.790520 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.790802 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.790790 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.843555 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:43.843519 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:43.843555 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.843557 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-16.us-west-2.compute.internal\": node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:43.952990 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:43.952905 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.053158 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.053113 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.154198 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.154153 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.200850 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.200822 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 08:31:44.201286 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.201002 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-16.us-west-2.compute.internal.18e3deab26576d6e\": read tcp 10.0.143.16:37854->3.151.158.10:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-16.us-west-2.compute.internal.18e3deab26576d6e default 5406 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-16.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-16.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-16.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 08:31:43 +0000 UTC,LastTimestamp:2026-11-01 08:31:43.441200197 +0000 UTC m=+0.530706475,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-16.us-west-2.compute.internal,}" Nov 01 08:31:44.254444 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.254379 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.355198 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.355157 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.455915 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.455877 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.468232 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.468197 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 08:26:44 +0000 UTC" deadline="2028-05-29 09:43:53.523258075 +0000 UTC" Nov 01 08:31:44.468232 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.468228 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13801h12m9.055032848s" Nov 01 08:31:44.544640 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.544561 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:31:44.545567 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.545551 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:31:44.545625 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.545583 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:31:44.545625 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.545595 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:31:44.549368 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.549349 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:31:44.556493 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:44.556472 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:31:44.594219 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.594184 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:31:44.594375 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.594254 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:31:44.666708 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.666670 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:44.738228 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.738187 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:31:44.738228 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.738216 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 08:31:44.739323 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.739303 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:31:44.796071 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.795992 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 08:31:44.959269 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:44.959222 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01f6a26e421fb5f2615ed4d9d59831f.slice/crio-ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74 WatchSource:0}: Error finding container ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74: Status 404 returned error can't find the container with id ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74 Nov 01 08:31:44.960229 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:44.960198 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61735862d7ea6a76be3182acee32ca86.slice/crio-92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b WatchSource:0}: Error finding container 92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b: Status 404 returned error can't find the container with id 92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b Nov 01 08:31:44.966818 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:44.966779 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 08:31:45.242206 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.242110 2580 apiserver.go:52] "Watching apiserver" Nov 01 08:31:45.266819 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.266784 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 08:31:45.345232 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.345178 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerStarted","Data":"ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74"} Nov 01 08:31:45.346428 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.346396 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"61735862d7ea6a76be3182acee32ca86","Type":"ContainerStarted","Data":"92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b"} Nov 01 08:31:45.346842 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.346826 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 08:31:45.349144 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.347629 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-gxwrb","openshift-network-diagnostics/network-check-target-z7b55","openshift-cluster-node-tuning-operator/tuned-fb5tm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-tg7cb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2","openshift-dns/node-resolver-4szqv","openshift-multus/multus-lxh88","kube-system/konnectivity-agent-t58md","openshift-multus/multus-additional-cni-plugins-8nckw","openshift-network-operator/iptables-alerter-2bmk2","openshift-ovn-kubernetes/ovnkube-node-jvnzm","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal","openshift-image-registry/node-ca-tg27q"] Nov 01 08:31:45.351993 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.351969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.354333 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.354299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.356643 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.356620 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.359056 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.359030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.361455 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.361431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.361566 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.361537 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:45.363754 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.363736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.366104 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.366084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.368439 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.368419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.370837 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.370778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:45.370920 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.370869 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:45.373093 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.373076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:45.373200 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.373179 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:45.375469 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.375454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.377980 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.377956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.381762 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.381863 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-systemd\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.381863 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f1c6a3-98c1-4970-9476-3330567de96c-serviceca\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.381940 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.381940 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-modprobe-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.381940 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-host\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.381940 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb599c9a-0381-45c2-801d-ef6578d0ea9d-host-slash\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.382075 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.381983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-etc-kubernetes\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382075 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-k8s-cni-cncf-io\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382075 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-hostroot\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4fh\" (UniqueName: \"kubernetes.io/projected/59c8058b-626b-4462-9765-5157b802d928-kube-api-access-6j4fh\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.382235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-tuned\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.382235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-dbus\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.382235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fld4z\" (UniqueName: \"kubernetes.io/projected/16f1c6a3-98c1-4970-9476-3330567de96c-kube-api-access-fld4z\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.382394 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-socket-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.382394 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhh4\" (UniqueName: \"kubernetes.io/projected/9ffc5727-de93-4b25-bd1c-46922c36a7e5-kube-api-access-cmhh4\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.382394 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382301 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7gx\" (UniqueName: \"kubernetes.io/projected/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kube-api-access-cn7gx\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.382394 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-cnibin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382394 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb599c9a-0381-45c2-801d-ef6578d0ea9d-iptables-alerter-script\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.382394 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-sys-fs\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.382572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-netns\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-conf-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57grl\" (UniqueName: \"kubernetes.io/projected/e97a3a69-d419-495c-877f-651eae6172f7-kube-api-access-57grl\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-system-cni-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.382572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-cni-binary-copy\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysconfig\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-kubernetes\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-multus\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-multus-daemon-config\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-multus-certs\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.382886 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-cnibin\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-kubelet\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-conf\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-sys\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f1c6a3-98c1-4970-9476-3330567de96c-host\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.382991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-registration-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-device-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-run\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-lib-modules\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-kubelet-config\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.383274 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-var-lib-kubelet\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzggv\" (UniqueName: \"kubernetes.io/projected/bb599c9a-0381-45c2-801d-ef6578d0ea9d-kube-api-access-qzggv\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-socket-dir-parent\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/958489ac-f598-4365-a166-22371ca307d4-konnectivity-ca\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-system-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-bin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-tmp\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.383741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.383539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/958489ac-f598-4365-a166-22371ca307d4-agent-certs\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.407521 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407444 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 08:31:45.407521 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407465 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.407521 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407486 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 08:31:45.407521 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407496 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.407521 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407467 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.407521 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407527 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xk5lv\"" Nov 01 08:31:45.407890 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407529 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 08:31:45.407890 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407556 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hnxt\"" Nov 01 08:31:45.407890 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.407557 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 08:31:45.410664 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.410633 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.468297 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.468252 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 08:31:45.484334 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:45.484334 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-tmp\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.484334 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/958489ac-f598-4365-a166-22371ca307d4-agent-certs\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76m8h\" (UniqueName: \"kubernetes.io/projected/7d57cfa0-eef3-450d-9030-3049ee009711-kube-api-access-76m8h\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-systemd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-etc-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-netd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-systemd\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.484671 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f1c6a3-98c1-4970-9476-3330567de96c-serviceca\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-modprobe-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484845 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-host\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-systemd\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb599c9a-0381-45c2-801d-ef6578d0ea9d-host-slash\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484904 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-modprobe-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484905 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-host\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-etc-kubernetes\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484929 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb599c9a-0381-45c2-801d-ef6578d0ea9d-host-slash\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-log-socket\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484955 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484952 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-etc-kubernetes\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.484989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-k8s-cni-cncf-io\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-hostroot\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485046 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-k8s-cni-cncf-io\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485072 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4fh\" (UniqueName: \"kubernetes.io/projected/59c8058b-626b-4462-9765-5157b802d928-kube-api-access-6j4fh\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-tuned\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485134 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-hostroot\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-dbus\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-netns\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fld4z\" (UniqueName: \"kubernetes.io/projected/16f1c6a3-98c1-4970-9476-3330567de96c-kube-api-access-fld4z\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-socket-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7rj\" (UniqueName: \"kubernetes.io/projected/388ff44e-1500-4858-87d0-7a33eee493d3-kube-api-access-5k7rj\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-kubelet\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485398 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-dbus\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhh4\" (UniqueName: \"kubernetes.io/projected/9ffc5727-de93-4b25-bd1c-46922c36a7e5-kube-api-access-cmhh4\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7gx\" (UniqueName: \"kubernetes.io/projected/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kube-api-access-cn7gx\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-cnibin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485502 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-socket-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-ovn\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-env-overrides\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb599c9a-0381-45c2-801d-ef6578d0ea9d-iptables-alerter-script\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.485949 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485597 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-cnibin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-config\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mln9l\" (UniqueName: \"kubernetes.io/projected/58067792-becb-4036-a80e-be1522d0f974-kube-api-access-mln9l\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-sys-fs\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-netns\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-conf-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485835 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-sys-fs\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57grl\" (UniqueName: \"kubernetes.io/projected/e97a3a69-d419-495c-877f-651eae6172f7-kube-api-access-57grl\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485863 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-netns\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485880 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-conf-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388ff44e-1500-4858-87d0-7a33eee493d3-hosts-file\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-system-cni-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.485997 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-system-cni-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.486081 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:45.486637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-cni-binary-copy\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.486164 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:31:45.986139126 +0000 UTC m=+3.075645410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-slash\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysconfig\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-kubernetes\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-kubernetes\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486324 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysconfig\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-multus\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-multus-daemon-config\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-multus-certs\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-systemd-units\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486558 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-multus\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-var-lib-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486625 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-multus-certs\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.487484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-cnibin\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-kubelet\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-kubelet\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-node-log\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-cnibin\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.486969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58067792-becb-4036-a80e-be1522d0f974-ovn-node-metrics-cert\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-conf\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-sys\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487079 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb599c9a-0381-45c2-801d-ef6578d0ea9d-iptables-alerter-script\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-sys\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f1c6a3-98c1-4970-9476-3330567de96c-host\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487159 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-conf\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487172 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-cni-binary-copy\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-registration-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.488179 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487214 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f1c6a3-98c1-4970-9476-3330567de96c-host\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-registration-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-device-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388ff44e-1500-4858-87d0-7a33eee493d3-tmp-dir\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-bin\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487345 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-device-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-run\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-lib-modules\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-kubelet-config\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-run\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487568 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-kubelet-config\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487569 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-script-lib\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487633 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-lib-modules\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.488676 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-var-lib-kubelet\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzggv\" (UniqueName: \"kubernetes.io/projected/bb599c9a-0381-45c2-801d-ef6578d0ea9d-kube-api-access-qzggv\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-socket-dir-parent\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487734 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-var-lib-kubelet\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487780 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-socket-dir-parent\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/958489ac-f598-4365-a166-22371ca307d4-konnectivity-ca\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-system-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-bin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487931 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-system-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487974 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.487989 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-bin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.488136 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.488322 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.489131 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-tmp\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.489318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.489135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-tuned\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.508581 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.508506 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 08:31:45.508703 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.508590 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s2jvz" Nov 01 08:31:45.520190 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520117 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.520190 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520117 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 08:31:45.520190 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520183 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 08:31:45.520470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520272 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.520470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520312 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.520470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520319 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.520470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520373 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.520470 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520381 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.520658 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520546 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d96l\"" Nov 01 08:31:45.520658 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520598 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 08:31:45.520658 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520622 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 08:31:45.520839 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520654 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 08:31:45.520839 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520668 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.520839 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520660 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 08:31:45.520839 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520681 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.520839 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.520652 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 08:31:45.525337 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.525312 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f1c6a3-98c1-4970-9476-3330567de96c-serviceca\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.528716 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528677 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 08:31:45.528716 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528689 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 08:31:45.528716 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528701 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqp75\"" Nov 01 08:31:45.528716 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528707 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48mlk\"" Nov 01 08:31:45.528716 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528689 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxggb\"" Nov 01 08:31:45.528994 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528687 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24cps\"" Nov 01 08:31:45.528994 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/958489ac-f598-4365-a166-22371ca307d4-konnectivity-ca\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.528994 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528836 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9vpz\"" Nov 01 08:31:45.528994 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.528871 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 08:31:45.537268 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.537228 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-multus-daemon-config\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.538142 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.538119 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/958489ac-f598-4365-a166-22371ca307d4-agent-certs\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.571318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.571290 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cdn26\"" Nov 01 08:31:45.571318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.571312 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 08:31:45.574382 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.574355 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4fh\" (UniqueName: \"kubernetes.io/projected/59c8058b-626b-4462-9765-5157b802d928-kube-api-access-6j4fh\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.574499 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.574355 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fld4z\" (UniqueName: \"kubernetes.io/projected/16f1c6a3-98c1-4970-9476-3330567de96c-kube-api-access-fld4z\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.582556 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.582524 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhh4\" (UniqueName: \"kubernetes.io/projected/9ffc5727-de93-4b25-bd1c-46922c36a7e5-kube-api-access-cmhh4\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.582556 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.582546 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzggv\" (UniqueName: \"kubernetes.io/projected/bb599c9a-0381-45c2-801d-ef6578d0ea9d-kube-api-access-qzggv\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.582723 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.582623 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57grl\" (UniqueName: \"kubernetes.io/projected/e97a3a69-d419-495c-877f-651eae6172f7-kube-api-access-57grl\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.582723 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.582653 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7gx\" (UniqueName: \"kubernetes.io/projected/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kube-api-access-cn7gx\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.588626 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-log-socket\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588702 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-netns\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588702 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7rj\" (UniqueName: \"kubernetes.io/projected/388ff44e-1500-4858-87d0-7a33eee493d3-kube-api-access-5k7rj\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.588702 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-kubelet\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588697 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-log-socket\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588708 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-netns\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-ovn\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588768 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-kubelet\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-env-overrides\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-config\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.588859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln9l\" (UniqueName: \"kubernetes.io/projected/58067792-becb-4036-a80e-be1522d0f974-kube-api-access-mln9l\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588810 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-ovn\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.588887 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.588889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388ff44e-1500-4858-87d0-7a33eee493d3-hosts-file\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.588990 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:46.088939419 +0000 UTC m=+3.178445706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-slash\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589100 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388ff44e-1500-4858-87d0-7a33eee493d3-hosts-file\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-systemd-units\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-var-lib-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-slash\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589193 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-systemd-units\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589208 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-node-log\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589223 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-var-lib-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.589235 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58067792-becb-4036-a80e-be1522d0f974-ovn-node-metrics-cert\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589267 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589248 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-node-log\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388ff44e-1500-4858-87d0-7a33eee493d3-tmp-dir\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589296 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-env-overrides\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-bin\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589340 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-bin\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-script-lib\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76m8h\" (UniqueName: \"kubernetes.io/projected/7d57cfa0-eef3-450d-9030-3049ee009711-kube-api-access-76m8h\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-systemd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-etc-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-netd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589498 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-config\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-netd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388ff44e-1500-4858-87d0-7a33eee493d3-tmp-dir\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.590013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-etc-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590493 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-systemd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.590493 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.589957 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-script-lib\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.591829 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.591811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58067792-becb-4036-a80e-be1522d0f974-ovn-node-metrics-cert\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.652133 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.652095 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:31:45.652133 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.652133 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:31:45.652315 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.652149 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:45.652315 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.652238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:46.152211142 +0000 UTC m=+3.241717429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:45.654795 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.654765 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln9l\" (UniqueName: \"kubernetes.io/projected/58067792-becb-4036-a80e-be1522d0f974-kube-api-access-mln9l\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.655287 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.655259 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7rj\" (UniqueName: \"kubernetes.io/projected/388ff44e-1500-4858-87d0-7a33eee493d3-kube-api-access-5k7rj\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.655441 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.655426 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76m8h\" (UniqueName: \"kubernetes.io/projected/7d57cfa0-eef3-450d-9030-3049ee009711-kube-api-access-76m8h\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:45.676985 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.676944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:31:45.679257 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.679220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:31:45.685965 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.685943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:31:45.686073 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.686030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:31:45.690485 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.690456 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb599c9a_0381_45c2_801d_ef6578d0ea9d.slice/crio-9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4 WatchSource:0}: Error finding container 9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4: Status 404 returned error can't find the container with id 9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4 Nov 01 08:31:45.691782 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.691711 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffc5727_de93_4b25_bd1c_46922c36a7e5.slice/crio-31040d5df38d1ac3512c0876792fe4d20d64ed652d8f79efb015156e203f6944 WatchSource:0}: Error finding container 31040d5df38d1ac3512c0876792fe4d20d64ed652d8f79efb015156e203f6944: Status 404 returned error can't find the container with id 31040d5df38d1ac3512c0876792fe4d20d64ed652d8f79efb015156e203f6944 Nov 01 08:31:45.695850 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.695823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:31:45.698315 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.698038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:45.698782 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.698740 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f1c6a3_98c1_4970_9476_3330567de96c.slice/crio-005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8 WatchSource:0}: Error finding container 005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8: Status 404 returned error can't find the container with id 005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8 Nov 01 08:31:45.704544 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.704504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxh88" Nov 01 08:31:45.707738 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.707702 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c8058b_626b_4462_9765_5157b802d928.slice/crio-02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a WatchSource:0}: Error finding container 02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a: Status 404 returned error can't find the container with id 02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a Nov 01 08:31:45.708992 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.708961 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07fa048_d1fa_4b73_8120_6d04c02c91e9.slice/crio-451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d WatchSource:0}: Error finding container 451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d: Status 404 returned error can't find the container with id 451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d Nov 01 08:31:45.712306 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.712279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4szqv" Nov 01 08:31:45.715008 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.714983 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958489ac_f598_4365_a166_22371ca307d4.slice/crio-aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb WatchSource:0}: Error finding container aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb: Status 404 returned error can't find the container with id aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb Nov 01 08:31:45.717583 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.717559 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97a3a69_d419_495c_877f_651eae6172f7.slice/crio-01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f WatchSource:0}: Error finding container 01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f: Status 404 returned error can't find the container with id 01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f Nov 01 08:31:45.723687 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.723649 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388ff44e_1500_4858_87d0_7a33eee493d3.slice/crio-96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89 WatchSource:0}: Error finding container 96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89: Status 404 returned error can't find the container with id 96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89 Nov 01 08:31:45.732449 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.732419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:31:45.743832 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:31:45.743787 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58067792_becb_4036_a80e_be1522d0f974.slice/crio-5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f WatchSource:0}: Error finding container 5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f: Status 404 returned error can't find the container with id 5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f Nov 01 08:31:45.992778 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:45.992639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:45.992942 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.992813 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:45.992942 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:45.992884 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:31:46.992865068 +0000 UTC m=+4.082371350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:46.093859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.093811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:46.094030 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:46.093974 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:46.094105 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:46.094042 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:47.094026028 +0000 UTC m=+4.183532310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:46.194933 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.194893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:46.195138 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:46.195043 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:31:46.195138 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:46.195068 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:31:46.195138 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:46.195084 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:46.195323 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:46.195153 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:47.195131489 +0000 UTC m=+4.284637772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:46.350288 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.350098 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" event={"ID":"9ffc5727-de93-4b25-bd1c-46922c36a7e5","Type":"ContainerStarted","Data":"31040d5df38d1ac3512c0876792fe4d20d64ed652d8f79efb015156e203f6944"} Nov 01 08:31:46.352471 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.352432 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f"} Nov 01 08:31:46.354964 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.354932 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxh88" event={"ID":"e97a3a69-d419-495c-877f-651eae6172f7","Type":"ContainerStarted","Data":"01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f"} Nov 01 08:31:46.357056 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.357021 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t58md" event={"ID":"958489ac-f598-4365-a166-22371ca307d4","Type":"ContainerStarted","Data":"aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb"} Nov 01 08:31:46.358444 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.358404 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tg27q" event={"ID":"16f1c6a3-98c1-4970-9476-3330567de96c","Type":"ContainerStarted","Data":"005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8"} Nov 01 08:31:46.359901 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.359863 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bmk2" event={"ID":"bb599c9a-0381-45c2-801d-ef6578d0ea9d","Type":"ContainerStarted","Data":"9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4"} Nov 01 08:31:46.361316 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.361287 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4szqv" event={"ID":"388ff44e-1500-4858-87d0-7a33eee493d3","Type":"ContainerStarted","Data":"96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89"} Nov 01 08:31:46.362950 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.362922 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" event={"ID":"a07fa048-d1fa-4b73-8120-6d04c02c91e9","Type":"ContainerStarted","Data":"451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d"} Nov 01 08:31:46.364462 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.364430 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerStarted","Data":"02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a"} Nov 01 08:31:46.513079 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.513004 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:46 +0000 UTC" deadline="2028-07-12 02:47:09.629593532 +0000 UTC" Nov 01 08:31:46.513079 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:46.513071 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14850h15m23.116526902s" Nov 01 08:31:47.000920 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.000872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:47.001131 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.001050 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:47.001131 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.001127 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:31:49.001105959 +0000 UTC m=+6.090612227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:47.102262 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.102167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:47.102447 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.102402 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:47.102511 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.102468 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:49.102449036 +0000 UTC m=+6.191955309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:47.203663 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.203272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:47.203663 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.203499 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:31:47.203663 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.203526 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:31:47.203663 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.203542 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:47.203663 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.203615 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:49.20359277 +0000 UTC m=+6.293099043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:47.341849 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.341433 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:47.341849 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.341598 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:47.341849 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.341696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:47.341849 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.341800 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:47.342427 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.342289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:47.342427 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:47.342391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:47.514383 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.514247 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:46 +0000 UTC" deadline="2028-06-10 00:32:58.863548007 +0000 UTC" Nov 01 08:31:47.514383 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:47.514322 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14080h1m11.349229895s" Nov 01 08:31:49.021494 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.021456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:49.021963 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.021613 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:49.021963 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.021664 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:31:53.021650309 +0000 UTC m=+10.111156580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:49.122563 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.122517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:49.122797 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.122721 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:49.122874 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.122817 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:53.122795619 +0000 UTC m=+10.212301901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:49.223854 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.223320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:49.223854 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.223569 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:31:49.223854 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.223595 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:31:49.223854 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.223610 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:49.223854 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.223678 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:31:53.223656622 +0000 UTC m=+10.313162892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:49.344481 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.340683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:49.344481 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.341143 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:49.344481 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.341286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:49.344481 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.341371 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:49.344481 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.341449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:49.344481 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:49.341557 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:49.421707 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:49.421632 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerStarted","Data":"39ac6264f44c61b1cf8899609bec5c4bd0dadcefb4422e81845348359ca190cd"} Nov 01 08:31:50.429628 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.428649 2580 generic.go:356] "Generic (PLEG): container finished" podID="a01f6a26e421fb5f2615ed4d9d59831f" containerID="39ac6264f44c61b1cf8899609bec5c4bd0dadcefb4422e81845348359ca190cd" exitCode=0 Nov 01 08:31:50.429628 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.428774 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerDied","Data":"39ac6264f44c61b1cf8899609bec5c4bd0dadcefb4422e81845348359ca190cd"} Nov 01 08:31:50.429628 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.428831 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerStarted","Data":"b898206dc5481bd1d3ed5c77c96d23fe6c08650f7faf489b0b552ae8e254ffcf"} Nov 01 08:31:50.433695 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.432787 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"61735862d7ea6a76be3182acee32ca86","Type":"ContainerStarted","Data":"f617abb16c2801874b0b61df6c5ee619e073fc5cce6e60b1e658dbcf7f009571"} Nov 01 08:31:50.437037 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.436759 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="83d637d063dac1935ac195375fc78294ef396de819530b4cc610ea4d70b37e63" exitCode=0 Nov 01 08:31:50.437037 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.436821 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerDied","Data":"83d637d063dac1935ac195375fc78294ef396de819530b4cc610ea4d70b37e63"} Nov 01 08:31:50.665544 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.665473 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" podStartSLOduration=5.665452278 podStartE2EDuration="5.665452278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:50.548619678 +0000 UTC m=+7.638125970" watchObservedRunningTime="2026-11-01 08:31:50.665452278 +0000 UTC m=+7.754958571" Nov 01 08:31:50.800383 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:50.800248 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" podStartSLOduration=5.800225462 podStartE2EDuration="5.800225462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:50.66625607 +0000 UTC m=+7.755762359" watchObservedRunningTime="2026-11-01 08:31:50.800225462 +0000 UTC m=+7.889731754" Nov 01 08:31:51.352215 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:51.351648 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:51.352215 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:51.351809 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:51.352215 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:51.351909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:51.352215 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:51.352003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:51.352215 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:51.352060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:51.352215 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:51.352129 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:51.443373 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:51.442587 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t58md" event={"ID":"958489ac-f598-4365-a166-22371ca307d4","Type":"ContainerStarted","Data":"eb787327f61898d8ddd89e6fbcd7a0a21888f34739247d2661948486cfd9900f"} Nov 01 08:31:52.448494 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:52.448365 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tg27q" event={"ID":"16f1c6a3-98c1-4970-9476-3330567de96c","Type":"ContainerStarted","Data":"be7fff7b500d1340d53e593b17615d06f14409806d232163ddd467bda9913bb5"} Nov 01 08:31:52.564171 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:52.564101 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t58md" podStartSLOduration=3.407927808 podStartE2EDuration="8.564079369s" totalImagesPullingTime="5.156151561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:51.596380347 +0000 UTC m=+8.685886640" watchObservedRunningTime="2026-11-01 08:31:52.564079369 +0000 UTC m=+9.653585669" Nov 01 08:31:53.059111 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.059067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:53.059337 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.059321 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:53.059404 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.059394 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:32:01.059372975 +0000 UTC m=+18.148879245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:31:53.160444 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.160389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:53.160732 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.160709 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:53.160908 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.160856 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:01.160817145 +0000 UTC m=+18.250323414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:31:53.261649 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.261554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:53.261850 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.261731 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:31:53.261850 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.261776 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:31:53.261850 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.261792 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:53.261984 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.261868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:01.261837871 +0000 UTC m=+18.351344157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:31:53.340628 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.340584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:53.340863 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.340729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:53.345016 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.341999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:53.345016 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.342119 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:53.345016 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.342198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:53.345016 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:53.342267 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:53.454255 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:53.454153 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" event={"ID":"a07fa048-d1fa-4b73-8120-6d04c02c91e9","Type":"ContainerStarted","Data":"4c22287cb08e1c737dc04f19ff0b538019eb38c48d619b6aed0ed72a426298d4"} Nov 01 08:31:55.345038 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.344986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:55.345525 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:55.345137 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:55.345525 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.345197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:55.345525 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:55.345247 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:55.345525 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.345286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:55.345525 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:55.345328 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:55.376893 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.376866 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 08:31:55.467840 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.467748 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" event={"ID":"a07fa048-d1fa-4b73-8120-6d04c02c91e9","Type":"ContainerStarted","Data":"0dd37c7ca812258c7b8dfce77948ec80239fba1b333464a9deb0ccfe10eb729d"} Nov 01 08:31:55.471046 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.471006 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxh88" event={"ID":"e97a3a69-d419-495c-877f-651eae6172f7","Type":"ContainerStarted","Data":"581bbfc102a99440cc208a0738edac347828e5b5fce8fbb3448b840994247091"} Nov 01 08:31:55.582400 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.582344 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tg27q" podStartSLOduration=5.303965497 podStartE2EDuration="11.582327695s" totalImagesPullingTime="6.278362198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:52.565242829 +0000 UTC m=+9.654749123" watchObservedRunningTime="2026-11-01 08:31:55.582327695 +0000 UTC m=+12.671833983" Nov 01 08:31:55.601799 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.601765 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:55.602594 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.602567 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:55.711110 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:55.711048 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lxh88" podStartSLOduration=2.709949348 podStartE2EDuration="11.711025969s" totalImagesPullingTime="9.001076621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:55.582944852 +0000 UTC m=+12.672451153" watchObservedRunningTime="2026-11-01 08:31:55.711025969 +0000 UTC m=+12.800532258" Nov 01 08:31:56.348991 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.348820 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T08:31:55.376887447Z","UUID":"aaaa2426-0072-4fbd-9628-423338585fba","Handler":null,"Name":"","Endpoint":""} Nov 01 08:31:56.350537 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.350510 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 08:31:56.350537 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.350547 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 08:31:56.475141 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.475098 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4szqv" event={"ID":"388ff44e-1500-4858-87d0-7a33eee493d3","Type":"ContainerStarted","Data":"ad3c286326125a77829b5bfd38e99da0cdc5c36f578211dd379b67308c50923f"} Nov 01 08:31:56.477602 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.477567 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="dd3f35704f3cdfd9bce341cd3b651ceb82e845a2cd2e0a1ac8251ba0d8955a23" exitCode=0 Nov 01 08:31:56.477901 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.477867 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerDied","Data":"dd3f35704f3cdfd9bce341cd3b651ceb82e845a2cd2e0a1ac8251ba0d8955a23"} Nov 01 08:31:56.698054 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:56.697575 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4szqv" podStartSLOduration=3.01862207 podStartE2EDuration="12.697559927s" totalImagesPullingTime="9.678937857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:56.583736906 +0000 UTC m=+13.673243188" watchObservedRunningTime="2026-11-01 08:31:56.697559927 +0000 UTC m=+13.787066214" Nov 01 08:31:57.343712 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.343674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:57.343919 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:57.343830 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:31:57.344004 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.343926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:57.344061 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:57.344003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:57.344114 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.344063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:57.344159 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:57.344129 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:57.406894 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.406848 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:57.408059 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.407981 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:31:57.482200 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.482164 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" event={"ID":"a07fa048-d1fa-4b73-8120-6d04c02c91e9","Type":"ContainerStarted","Data":"ac481d96fa74d2cc4a4a73c526c46d25a68f412613b21c9d08f41eda7762700e"} Nov 01 08:31:57.484472 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.484418 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" event={"ID":"9ffc5727-de93-4b25-bd1c-46922c36a7e5","Type":"ContainerStarted","Data":"bdabfda2cdbb72d5dc7faa9a3bb5008c27d3d15dbcaf61606d8bad578f0702df"} Nov 01 08:31:57.486048 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.486022 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bmk2" event={"ID":"bb599c9a-0381-45c2-801d-ef6578d0ea9d","Type":"ContainerStarted","Data":"88bf6d83977d1aa9c6a4214f0f74ec1e0e2eda3d3db836ab9bd23a928df3c1c0"} Nov 01 08:31:57.626703 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.626639 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" podStartSLOduration=2.14509754 podStartE2EDuration="13.626620587s" totalImagesPullingTime="11.481523047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:57.626055975 +0000 UTC m=+14.715562264" watchObservedRunningTime="2026-11-01 08:31:57.626620587 +0000 UTC m=+14.716126876" Nov 01 08:31:57.737986 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.737921 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" podStartSLOduration=2.711503235 podStartE2EDuration="13.737900908s" totalImagesPullingTime="11.026397673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:57.737298956 +0000 UTC m=+14.826805245" watchObservedRunningTime="2026-11-01 08:31:57.737900908 +0000 UTC m=+14.827407197" Nov 01 08:31:57.845448 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:57.845385 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2bmk2" podStartSLOduration=4.265765828 podStartE2EDuration="13.8453674s" totalImagesPullingTime="9.579601572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:31:57.844935766 +0000 UTC m=+14.934442054" watchObservedRunningTime="2026-11-01 08:31:57.8453674 +0000 UTC m=+14.934873687" Nov 01 08:31:58.490645 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:58.490598 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="82e4b379e2a3b4bcf8aa728afdd469bf9c654d4e3dc9c2122bb43b92a5281a1a" exitCode=0 Nov 01 08:31:58.491279 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:58.490682 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerDied","Data":"82e4b379e2a3b4bcf8aa728afdd469bf9c654d4e3dc9c2122bb43b92a5281a1a"} Nov 01 08:31:59.340589 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:59.340551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:31:59.340801 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:59.340551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:31:59.340801 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:59.340695 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:31:59.340801 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:31:59.340730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:31:59.340930 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:59.340854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:31:59.341013 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:31:59.340983 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:00.498030 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:00.497997 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="c35541d399cf4784c584f3059c263a1401ba14194baabe6b52e83c03f80ef059" exitCode=0 Nov 01 08:32:00.498784 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:00.498040 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerDied","Data":"c35541d399cf4784c584f3059c263a1401ba14194baabe6b52e83c03f80ef059"} Nov 01 08:32:01.133390 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:01.133349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:01.133629 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.133508 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:32:01.133629 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.133572 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:32:17.133555264 +0000 UTC m=+34.223061547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:32:01.234413 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:01.234371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:01.234587 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.234542 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:32:01.234635 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.234617 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:17.234599268 +0000 UTC m=+34.324105534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:32:01.335176 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:01.335138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:01.335335 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.335308 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:32:01.335335 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.335327 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:32:01.335335 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.335336 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:32:01.335477 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.335391 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:17.335378259 +0000 UTC m=+34.424884541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:32:01.344591 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:01.344561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:01.344798 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:01.344597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:01.344798 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:01.344605 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:01.344798 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.344733 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:01.344938 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.344844 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:01.344985 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:01.344929 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:02.505036 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:02.504808 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:02.505499 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:02.505414 2580 generic.go:356] "Generic (PLEG): container finished" podID="58067792-becb-4036-a80e-be1522d0f974" containerID="42c8d9328a047262c4f49963d221f8e96d9151b8d828319424e44f0d0e30c7f3" exitCode=1 Nov 01 08:32:02.505557 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:02.505488 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"d3efb8cd9f02b0d9b994261f8fd1a28ff756566fba2dc464a3fb1f12e2fdfbf5"} Nov 01 08:32:02.505608 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:02.505574 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerDied","Data":"42c8d9328a047262c4f49963d221f8e96d9151b8d828319424e44f0d0e30c7f3"} Nov 01 08:32:02.505649 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:02.505631 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"3eef5e3398b036d33fbd88c8a057a7c6db03fe275469f54665957469ff9b7a04"} Nov 01 08:32:03.341700 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.341439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:03.341863 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.341504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:03.341863 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:03.341753 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:03.341863 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:03.341811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:03.341863 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.341524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:03.342020 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:03.341896 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:03.510984 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.510941 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="332c3cd0cfe6ade73967c91aad905ee4f6517b42b566874aa7eaf92ddf14f666" exitCode=0 Nov 01 08:32:03.511518 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.511023 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerDied","Data":"332c3cd0cfe6ade73967c91aad905ee4f6517b42b566874aa7eaf92ddf14f666"} Nov 01 08:32:03.513977 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.513954 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:03.514335 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.514309 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"a2589c30d22a39e451dd107176eedf9bbb10b6b5d3c75b80d503a95201cd9585"} Nov 01 08:32:03.514390 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.514378 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"6162ad24bb57fcd255fe95c458a5c6b9f42a9418aab844a96abedc85702ace80"} Nov 01 08:32:03.514435 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:03.514412 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"7afaaa151f46fae98fd214abb95888aec0bc794eadff00fb58128418d1272893"} Nov 01 08:32:04.519511 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:04.519477 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="e7e3520c0f537b932e74b29471c723cdcce6509f0d3b0261b25cd7ec0f260181" exitCode=0 Nov 01 08:32:04.520007 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:04.519532 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerDied","Data":"e7e3520c0f537b932e74b29471c723cdcce6509f0d3b0261b25cd7ec0f260181"} Nov 01 08:32:05.344141 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.344111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:05.344324 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.344181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:05.344324 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.344209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:05.344324 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:05.344290 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:05.344324 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:05.344321 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:05.344502 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:05.344432 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:05.525174 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.525150 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:05.525817 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.525598 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"457eb838c0638b44e073482fc49b430948867e0e597cbf416401f66f99915229"} Nov 01 08:32:05.528998 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.528970 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8nckw" event={"ID":"59c8058b-626b-4462-9765-5157b802d928","Type":"ContainerStarted","Data":"c141ca6f9c6d3f3c55b08c53d490486db1840cf5111e546d81863c005aaaf734"} Nov 01 08:32:05.644547 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:05.644455 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8nckw" podStartSLOduration=7.800723795 podStartE2EDuration="21.644441731s" totalImagesPullingTime="13.843717936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:05.644233438 +0000 UTC m=+22.733739730" watchObservedRunningTime="2026-11-01 08:32:05.644441731 +0000 UTC m=+22.733948019" Nov 01 08:32:07.344262 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.344233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:07.344915 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.344333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:07.344915 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:07.344340 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:07.344915 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.344360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:07.344915 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:07.344431 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:07.344915 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:07.344511 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:07.357114 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.357076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t58md_958489ac-f598-4365-a166-22371ca307d4/konnectivity-agent/0.log" limit=1024 Nov 01 08:32:07.537637 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.537609 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:07.537976 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.537954 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"87ccc748729378a63c39e08b54853e0a7fafa52c0ef77fa62ef77af87b24ce05"} Nov 01 08:32:07.538225 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.538203 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:07.538290 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.538279 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:07.538400 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.538385 2580 scope.go:122] "RemoveContainer" containerID="42c8d9328a047262c4f49963d221f8e96d9151b8d828319424e44f0d0e30c7f3" Nov 01 08:32:07.555538 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.555510 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:07.565323 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:07.565299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t58md_958489ac-f598-4365-a166-22371ca307d4/konnectivity-agent/0.log" limit=1024 Nov 01 08:32:08.544263 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:08.544080 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:08.544742 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:08.544592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" event={"ID":"58067792-becb-4036-a80e-be1522d0f974","Type":"ContainerStarted","Data":"f1e7fad7f41625fbb40b3ad274519ef0574106f05539df23ea1fe14e3e1bd12e"} Nov 01 08:32:08.544950 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:08.544926 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:08.560969 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:08.560939 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:08.717405 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:08.717353 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" podStartSLOduration=8.249094716 podStartE2EDuration="24.717339261s" totalImagesPullingTime="16.468244545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:08.716362053 +0000 UTC m=+25.805868340" watchObservedRunningTime="2026-11-01 08:32:08.717339261 +0000 UTC m=+25.806845606" Nov 01 08:32:09.341106 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.341074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:09.341106 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.341094 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:09.341349 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:09.341224 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:09.341349 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.341299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:09.341435 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:09.341290 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:09.341435 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:09.341397 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:09.850943 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.850908 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tg7cb"] Nov 01 08:32:09.851447 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.851024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:09.851447 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:09.851148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:09.853660 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.853636 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7b55"] Nov 01 08:32:09.853827 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.853810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:09.853981 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:09.853948 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:09.854435 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.854412 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gxwrb"] Nov 01 08:32:09.854540 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:09.854507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:09.854638 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:09.854610 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:11.341014 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:11.340982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:11.341014 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:11.340979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:11.341505 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:11.341089 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:11.341505 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:11.341452 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:11.344605 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:11.344583 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:11.345085 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:11.345064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:13.341312 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:13.341272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:13.341312 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:13.341297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:13.342023 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:13.341279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:13.342023 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:13.341406 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:13.342023 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:13.341490 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:13.342023 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:13.341541 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:15.188935 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:15.188904 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4szqv_388ff44e-1500-4858-87d0-7a33eee493d3/dns-node-resolver/0.log" Nov 01 08:32:15.341200 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:15.341166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:15.341375 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:15.341196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:15.341375 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:15.341340 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:15.341467 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:15.341378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:15.341467 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:15.341445 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:15.341569 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:15.341552 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:16.589244 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:16.589221 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tg27q_16f1c6a3-98c1-4970-9476-3330567de96c/node-ca/0.log" Nov 01 08:32:17.149834 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:17.149779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:17.150094 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.149942 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 08:32:17.150094 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.150008 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret podName:fc68d9cc-01a2-49b3-b0f6-f510d04caefa nodeName:}" failed. No retries permitted until 2026-11-01 08:32:49.149990737 +0000 UTC m=+66.239497004 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret") pod "global-pull-secret-syncer-tg7cb" (UID: "fc68d9cc-01a2-49b3-b0f6-f510d04caefa") : object "kube-system"/"original-pull-secret" not registered Nov 01 08:32:17.251027 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:17.250983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:17.251213 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.251141 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:32:17.251213 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.251211 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs podName:7d57cfa0-eef3-450d-9030-3049ee009711 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:49.251195763 +0000 UTC m=+66.340702029 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs") pod "network-metrics-daemon-gxwrb" (UID: "7d57cfa0-eef3-450d-9030-3049ee009711") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 08:32:17.340892 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:17.340854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:17.341069 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:17.340854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:17.341069 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.340980 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:17.341192 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.341076 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:17.341192 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:17.340860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:17.341192 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.341177 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:17.352232 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:17.352195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:17.352431 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.352341 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 08:32:17.352431 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.352356 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 08:32:17.352431 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.352369 2580 projected.go:196] Error preparing data for projected volume kube-api-access-rd87k for pod openshift-network-diagnostics/network-check-target-z7b55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:32:17.352431 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:17.352422 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k podName:af1dcebb-bbfe-4e4a-b684-84012ddc57b8 nodeName:}" failed. No retries permitted until 2026-11-01 08:32:49.352408695 +0000 UTC m=+66.441914974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rd87k" (UniqueName: "kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k") pod "network-check-target-z7b55" (UID: "af1dcebb-bbfe-4e4a-b684-84012ddc57b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 08:32:19.340454 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:19.340429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:19.340929 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:19.340498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:19.340929 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:19.340502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:19.340929 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:19.340597 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:19.340929 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:19.340653 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:19.340929 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:19.340749 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:21.340982 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:21.340947 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:21.340982 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:21.340982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:21.341439 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:21.340947 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:21.341439 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:21.341084 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:21.341439 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:21.341131 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:21.341439 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:21.341242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:23.341130 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:23.341093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:23.341527 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:23.341105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:23.341527 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:23.341206 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:23.341527 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:23.341216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:23.341527 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:23.341320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:23.341527 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:23.341413 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:25.341008 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:25.340976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:25.341008 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:25.340994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:25.341453 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:25.340994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:25.341453 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:25.341075 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:25.341453 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:25.341175 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:25.341453 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:25.341244 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:27.341117 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:27.341086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:27.341516 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:27.341119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:27.341516 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:27.341191 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:27.341516 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:27.341271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:27.341516 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:27.341369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:27.341516 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:27.341498 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:29.341196 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:29.341164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:29.341196 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:29.341179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:29.341673 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:29.341276 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7b55" podUID="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" Nov 01 08:32:29.341673 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:29.341351 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gxwrb" podUID="7d57cfa0-eef3-450d-9030-3049ee009711" Nov 01 08:32:29.341673 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:29.341376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:29.341673 ip-10-0-143-16 kubenswrapper[2580]: E1101 08:32:29.341468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg7cb" podUID="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" Nov 01 08:32:30.715022 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.714938 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeReady" Nov 01 08:32:30.715492 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.715087 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 08:32:30.871456 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.871416 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bm9bj"] Nov 01 08:32:30.893589 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.893557 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5t549"] Nov 01 08:32:30.893779 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.893692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:30.908819 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.908792 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5t549"] Nov 01 08:32:30.908951 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.908826 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bm9bj"] Nov 01 08:32:30.908951 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.908909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:30.915074 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.915044 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9tkwt"] Nov 01 08:32:30.928963 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.928935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:30.942789 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.942760 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 08:32:30.942789 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.942790 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 08:32:30.942962 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.942802 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 08:32:30.943644 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxql\" (UniqueName: \"kubernetes.io/projected/29716515-5846-432e-b2a2-e63ce1ea7059-kube-api-access-ppxql\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:30.943704 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15ee2bde-64f4-4580-8969-c395abf4c2f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:30.943704 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:30.943704 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8nb\" (UniqueName: \"kubernetes.io/projected/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-api-access-kb8nb\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:30.943852 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29716515-5846-432e-b2a2-e63ce1ea7059-cert\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:30.943891 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15ee2bde-64f4-4580-8969-c395abf4c2f0-crio-socket\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:30.943891 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15ee2bde-64f4-4580-8969-c395abf4c2f0-data-volume\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:30.943961 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-metrics-tls\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:30.943998 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-config-volume\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:30.944032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.943995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8mx\" (UniqueName: \"kubernetes.io/projected/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-kube-api-access-4b8mx\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:30.944032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.944019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-tmp-dir\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:30.950479 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.950460 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 08:32:30.950599 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.950536 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmhw9\"" Nov 01 08:32:30.972782 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:30.972693 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9tkwt"] Nov 01 08:32:31.002769 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.002733 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 08:32:31.002962 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.002908 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mnx4s\"" Nov 01 08:32:31.002962 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.002917 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h4rfg\"" Nov 01 08:32:31.003050 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.002984 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 08:32:31.043253 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.043227 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 08:32:31.043253 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.043242 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 08:32:31.043253 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.043227 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 08:32:31.044631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15ee2bde-64f4-4580-8969-c395abf4c2f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.044692 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.044692 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8nb\" (UniqueName: \"kubernetes.io/projected/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-api-access-kb8nb\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.044798 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29716515-5846-432e-b2a2-e63ce1ea7059-cert\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:31.044798 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15ee2bde-64f4-4580-8969-c395abf4c2f0-crio-socket\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.044798 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15ee2bde-64f4-4580-8969-c395abf4c2f0-data-volume\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.044798 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-metrics-tls\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.044997 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044814 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15ee2bde-64f4-4580-8969-c395abf4c2f0-crio-socket\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.044997 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-config-volume\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.044997 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8mx\" (UniqueName: \"kubernetes.io/projected/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-kube-api-access-4b8mx\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.044997 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-tmp-dir\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.044997 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.044962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxql\" (UniqueName: \"kubernetes.io/projected/29716515-5846-432e-b2a2-e63ce1ea7059-kube-api-access-ppxql\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:31.045388 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.045358 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15ee2bde-64f4-4580-8969-c395abf4c2f0-data-volume\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.045532 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.045372 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-tmp-dir\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.045532 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.045478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.045532 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.045483 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-config-volume\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.048679 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.048659 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-metrics-tls\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.048791 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.048755 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15ee2bde-64f4-4580-8969-c395abf4c2f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.048859 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.048828 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29716515-5846-432e-b2a2-e63ce1ea7059-cert\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:31.102357 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.102324 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8nb\" (UniqueName: \"kubernetes.io/projected/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-api-access-kb8nb\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.102543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.102396 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8mx\" (UniqueName: \"kubernetes.io/projected/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-kube-api-access-4b8mx\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.102543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.102492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxql\" (UniqueName: \"kubernetes.io/projected/29716515-5846-432e-b2a2-e63ce1ea7059-kube-api-access-ppxql\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:31.251411 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.251324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:31.279560 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.279527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:31.301322 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.301299 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:31.341197 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.341170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:31.341197 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.341195 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:31.341329 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.341186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:31.390489 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390454 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 08:32:31.390672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390557 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 08:32:31.390672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390607 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 08:32:31.390672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390609 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\"" Nov 01 08:32:31.390672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390631 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\"" Nov 01 08:32:31.390672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390645 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\"" Nov 01 08:32:31.390925 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.390690 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 08:32:31.772322 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.771334 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5t549"] Nov 01 08:32:31.775452 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.775424 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bm9bj"] Nov 01 08:32:31.776298 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:31.776279 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9tkwt"] Nov 01 08:32:31.779407 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:31.779383 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29716515_5846_432e_b2a2_e63ce1ea7059.slice/crio-760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846 WatchSource:0}: Error finding container 760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846: Status 404 returned error can't find the container with id 760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846 Nov 01 08:32:31.782268 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:31.782239 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6611ffa7_eb5a_4a4f_be97_2d444e688e8d.slice/crio-7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc WatchSource:0}: Error finding container 7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc: Status 404 returned error can't find the container with id 7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc Nov 01 08:32:31.784940 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:31.783138 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ee2bde_64f4_4580_8969_c395abf4c2f0.slice/crio-e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31 WatchSource:0}: Error finding container e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31: Status 404 returned error can't find the container with id e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31 Nov 01 08:32:32.121632 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.121603 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6"] Nov 01 08:32:32.132953 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.132921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:32.154418 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.154377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0c527af-be39-4887-990a-0dd793e2a751-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2j2r6\" (UID: \"d0c527af-be39-4887-990a-0dd793e2a751\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:32.178110 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.178078 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6"] Nov 01 08:32:32.181908 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.181885 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 08:32:32.182070 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.182056 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lnbwc\"" Nov 01 08:32:32.254771 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.254711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0c527af-be39-4887-990a-0dd793e2a751-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2j2r6\" (UID: \"d0c527af-be39-4887-990a-0dd793e2a751\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:32.257371 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.257342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0c527af-be39-4887-990a-0dd793e2a751-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2j2r6\" (UID: \"d0c527af-be39-4887-990a-0dd793e2a751\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:32.451066 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.450975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:32.600508 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.600476 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm9bj" event={"ID":"15ee2bde-64f4-4580-8969-c395abf4c2f0","Type":"ContainerStarted","Data":"31a90f1e240569d1f0b028f9cc26da4aaf3310b02ec1bd69901fdd9a7c48be4b"} Nov 01 08:32:32.600672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.600534 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm9bj" event={"ID":"15ee2bde-64f4-4580-8969-c395abf4c2f0","Type":"ContainerStarted","Data":"e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31"} Nov 01 08:32:32.601697 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.601673 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5t549" event={"ID":"29716515-5846-432e-b2a2-e63ce1ea7059","Type":"ContainerStarted","Data":"760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846"} Nov 01 08:32:32.602836 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.602815 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tkwt" event={"ID":"6611ffa7-eb5a-4a4f-be97-2d444e688e8d","Type":"ContainerStarted","Data":"7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc"} Nov 01 08:32:32.827317 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:32.827218 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6"] Nov 01 08:32:32.835284 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:32.835258 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c527af_be39_4887_990a_0dd793e2a751.slice/crio-36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02 WatchSource:0}: Error finding container 36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02: Status 404 returned error can't find the container with id 36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02 Nov 01 08:32:33.369968 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.369929 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b9c7fbff-kdtfn"] Nov 01 08:32:33.395396 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.395362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.429128 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.429090 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b9c7fbff-kdtfn"] Nov 01 08:32:33.464269 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.464567 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.464567 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.464567 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.464769 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.464769 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.464769 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.464755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.474268 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.474236 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 08:32:33.474475 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.474275 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8k6cl\"" Nov 01 08:32:33.474958 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.474935 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 08:32:33.491084 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.491049 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 08:32:33.491272 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.491169 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 08:32:33.491272 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.491247 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 08:32:33.491378 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.491278 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 08:32:33.520832 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.520802 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 08:32:33.565870 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.565827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566063 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.565886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566063 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.565914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566063 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.565939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566063 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.565980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566063 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.566007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566063 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.566049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.566943 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.566907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.567300 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.567237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.567588 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.567567 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.568013 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.567991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.570037 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.570011 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.570751 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.570167 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.606916 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.606878 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" event={"ID":"d0c527af-be39-4887-990a-0dd793e2a751","Type":"ContainerStarted","Data":"36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02"} Nov 01 08:32:33.625916 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.625828 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:33.723193 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:33.722940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:34.145741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:34.145686 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b9c7fbff-kdtfn"] Nov 01 08:32:34.183372 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:34.183339 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0580a40d_15af_4a92_86d4_eb3fa865b8a8.slice/crio-2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692 WatchSource:0}: Error finding container 2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692: Status 404 returned error can't find the container with id 2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692 Nov 01 08:32:34.611660 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:34.611619 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm9bj" event={"ID":"15ee2bde-64f4-4580-8969-c395abf4c2f0","Type":"ContainerStarted","Data":"7c5116aa422fda120604681cef9d880a64588135b3a7a03bdc3aee853733974c"} Nov 01 08:32:34.612968 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:34.612944 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b9c7fbff-kdtfn" event={"ID":"0580a40d-15af-4a92-86d4-eb3fa865b8a8","Type":"ContainerStarted","Data":"2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692"} Nov 01 08:32:35.618659 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:35.618620 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tkwt" event={"ID":"6611ffa7-eb5a-4a4f-be97-2d444e688e8d","Type":"ContainerStarted","Data":"3a76d41eb24302d27b9c706123a920046deaa08f5f31ceec3c861827faba39b2"} Nov 01 08:32:35.619225 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:35.618679 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9tkwt" event={"ID":"6611ffa7-eb5a-4a4f-be97-2d444e688e8d","Type":"ContainerStarted","Data":"b2f7ec687ec9416e9619d2eb9c68b98133048155ba5cc8bd95d523318bc54aa9"} Nov 01 08:32:35.619546 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:35.619526 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:35.730789 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:35.730705 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9tkwt" podStartSLOduration=2.663096596 podStartE2EDuration="5.730687325s" totalImagesPullingTime="3.067590729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:35.729531172 +0000 UTC m=+52.819037462" watchObservedRunningTime="2026-11-01 08:32:35.730687325 +0000 UTC m=+52.820193613" Nov 01 08:32:36.623568 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:36.623533 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5t549" event={"ID":"29716515-5846-432e-b2a2-e63ce1ea7059","Type":"ContainerStarted","Data":"18fac12d9f6da83db026328a36e4a0073773297bda3ae57f0f331a9358641052"} Nov 01 08:32:36.625320 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:36.625298 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" event={"ID":"d0c527af-be39-4887-990a-0dd793e2a751","Type":"ContainerStarted","Data":"d485312962fae548592a4baa65944e4c5fd79aed48620efee0290644a77950a0"} Nov 01 08:32:36.625614 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:36.625593 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:36.632809 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:36.632783 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:36.732509 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:36.732458 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5t549" podStartSLOduration=2.242949071 podStartE2EDuration="6.732443025s" totalImagesPullingTime="4.489493954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:36.731881931 +0000 UTC m=+53.821388220" watchObservedRunningTime="2026-11-01 08:32:36.732443025 +0000 UTC m=+53.821949313" Nov 01 08:32:36.841134 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:36.840647 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" podStartSLOduration=2.037283176 podStartE2EDuration="4.840626658s" totalImagesPullingTime="2.803343482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:36.840032643 +0000 UTC m=+53.929538932" watchObservedRunningTime="2026-11-01 08:32:36.840626658 +0000 UTC m=+53.930132947" Nov 01 08:32:37.630760 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.630690 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm9bj" event={"ID":"15ee2bde-64f4-4580-8969-c395abf4c2f0","Type":"ContainerStarted","Data":"4c120b05092baf866813c086643cf5b74572bc6806a612e33235449334efe9cc"} Nov 01 08:32:37.729778 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.729711 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vssf9"] Nov 01 08:32:37.732704 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.732687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.776212 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.776160 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bm9bj" podStartSLOduration=3.195934325 podStartE2EDuration="7.776145955s" totalImagesPullingTime="4.58021163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 08:32:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:37.744735269 +0000 UTC m=+54.834241553" watchObservedRunningTime="2026-11-01 08:32:37.776145955 +0000 UTC m=+54.865652242" Nov 01 08:32:37.777417 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.777399 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vssf9"] Nov 01 08:32:37.791245 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.791212 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 08:32:37.791385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.791252 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 08:32:37.791385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.791270 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 08:32:37.791385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.791277 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 08:32:37.791385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.791272 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 08:32:37.791385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.791300 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6hfzp\"" Nov 01 08:32:37.799414 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.799382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/22098de3-a796-47bb-98c9-30991caa95ac-kube-api-access-mnx4n\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.799475 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.799453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.799523 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.799476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22098de3-a796-47bb-98c9-30991caa95ac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.799600 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.799577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.900347 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.900243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.900347 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.900289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22098de3-a796-47bb-98c9-30991caa95ac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.900580 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.900353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.900580 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.900390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/22098de3-a796-47bb-98c9-30991caa95ac-kube-api-access-mnx4n\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.901143 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.901119 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22098de3-a796-47bb-98c9-30991caa95ac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.903343 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.903318 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.903457 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.903388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:37.960734 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:37.960695 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/22098de3-a796-47bb-98c9-30991caa95ac-kube-api-access-mnx4n\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:38.047906 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:38.047872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:38.415801 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:38.415764 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vssf9"] Nov 01 08:32:38.424265 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:38.424232 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22098de3_a796_47bb_98c9_30991caa95ac.slice/crio-b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f WatchSource:0}: Error finding container b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f: Status 404 returned error can't find the container with id b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f Nov 01 08:32:38.636391 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:38.636351 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" event={"ID":"22098de3-a796-47bb-98c9-30991caa95ac","Type":"ContainerStarted","Data":"b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f"} Nov 01 08:32:40.563134 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:40.563101 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:41.647389 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:41.647349 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b9c7fbff-kdtfn" event={"ID":"0580a40d-15af-4a92-86d4-eb3fa865b8a8","Type":"ContainerStarted","Data":"7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e"} Nov 01 08:32:41.759731 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:41.759660 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8b9c7fbff-kdtfn" podStartSLOduration=1.801857099 podStartE2EDuration="8.759643954s" totalImagesPullingTime="6.957786855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:41.759356213 +0000 UTC m=+58.848862501" watchObservedRunningTime="2026-11-01 08:32:41.759643954 +0000 UTC m=+58.849150238" Nov 01 08:32:42.652708 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:42.652664 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" event={"ID":"22098de3-a796-47bb-98c9-30991caa95ac","Type":"ContainerStarted","Data":"15b6ca1b6689f34c91c4ff1bba6e4789732abe5452174e32f49aef0fb5ba62d4"} Nov 01 08:32:42.653151 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:42.652742 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" event={"ID":"22098de3-a796-47bb-98c9-30991caa95ac","Type":"ContainerStarted","Data":"b8660ec022a69f51cb231c06ef16e60029818746170d02cb70fbc39b0942b60d"} Nov 01 08:32:42.795540 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:42.795435 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" podStartSLOduration=1.7985713589999999 podStartE2EDuration="5.795419777s" totalImagesPullingTime="3.996848418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:42.795027316 +0000 UTC m=+59.884533604" watchObservedRunningTime="2026-11-01 08:32:42.795419777 +0000 UTC m=+59.884926065" Nov 01 08:32:43.724226 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:43.724187 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:43.724664 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:43.724487 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:43.729223 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:43.729198 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:44.662449 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:44.662422 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:45.614379 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.614340 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66fbd6bf9f-hrn97"] Nov 01 08:32:45.619378 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.619347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660029 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.659992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660029 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660234 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660234 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660304 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660347 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660347 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.660450 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.660424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.761573 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.761532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.761816 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.761585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.761961 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.761939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762026 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762079 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762079 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762177 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762087 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762177 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762384 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762356 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 08:32:45.762556 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.762903 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.762857 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.763080 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.763054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.763270 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.763249 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.763764 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.763742 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fbd6bf9f-hrn97"] Nov 01 08:32:45.764594 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.764570 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.764680 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.764576 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.773229 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.773205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.820915 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.820876 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:45.929847 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:45.929751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:46.033297 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.033253 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5"] Nov 01 08:32:46.037642 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.037616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.065408 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.065376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.065572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.065426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx75b\" (UniqueName: \"kubernetes.io/projected/ee31b8bb-4ebe-41dd-a701-66eaf42137da-kube-api-access-cx75b\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.065572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.065554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.065672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.065587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee31b8bb-4ebe-41dd-a701-66eaf42137da-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.086518 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.086488 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 08:32:46.086635 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.086569 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 08:32:46.086635 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.086590 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sctgh\"" Nov 01 08:32:46.095551 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.095518 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5"] Nov 01 08:32:46.138306 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.138269 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p7p9g"] Nov 01 08:32:46.142042 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.142015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.145205 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.145177 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc"] Nov 01 08:32:46.148572 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.148546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.166862 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.166829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gbk5\" (UniqueName: \"kubernetes.io/projected/7a28283c-3f57-4276-a7d1-a7792377f37b-kube-api-access-8gbk5\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.166873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-root\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.166902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.167032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.166925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.166979 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-tls\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167032 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx75b\" (UniqueName: \"kubernetes.io/projected/ee31b8bb-4ebe-41dd-a701-66eaf42137da-kube-api-access-cx75b\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.167214 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-wtmp\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167214 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.167214 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee31b8bb-4ebe-41dd-a701-66eaf42137da-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.167351 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-sys\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167351 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-textfile\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167351 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167510 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-metrics-client-ca\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.167888 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.167869 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee31b8bb-4ebe-41dd-a701-66eaf42137da-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.169477 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.169458 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.169633 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.169616 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.191139 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.191056 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 08:32:46.191302 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.191211 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 08:32:46.191302 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.191269 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 08:32:46.191302 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.191293 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qwt9q\"" Nov 01 08:32:46.204279 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.204244 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc"] Nov 01 08:32:46.224442 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.224417 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx75b\" (UniqueName: \"kubernetes.io/projected/ee31b8bb-4ebe-41dd-a701-66eaf42137da-kube-api-access-cx75b\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.267920 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.267889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbk5\" (UniqueName: \"kubernetes.io/projected/7a28283c-3f57-4276-a7d1-a7792377f37b-kube-api-access-8gbk5\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.267920 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.267920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-root\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268133 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.267954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.268133 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.267986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.268133 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268133 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268052 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-root\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268133 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-tls\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268133 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-wtmp\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268365 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.268365 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ljf\" (UniqueName: \"kubernetes.io/projected/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-api-access-d6ljf\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.268365 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-sys\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268365 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-wtmp\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268365 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.268543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-textfile\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-metrics-client-ca\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268543 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.268688 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268617 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-sys\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268688 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268624 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.268865 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.268831 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-textfile\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.269052 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.269033 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-metrics-client-ca\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.270706 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.270681 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-tls\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.271275 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.271259 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.292043 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.292017 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 08:32:46.292201 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.292122 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 08:32:46.292201 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.292164 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d5rrp\"" Nov 01 08:32:46.292201 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.292174 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 08:32:46.301963 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.301934 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fbd6bf9f-hrn97"] Nov 01 08:32:46.310473 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:46.310441 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4e88fb_f709_4c95_828d_9ef46ff8e159.slice/crio-3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871 WatchSource:0}: Error finding container 3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871: Status 404 returned error can't find the container with id 3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871 Nov 01 08:32:46.325602 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.325579 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbk5\" (UniqueName: \"kubernetes.io/projected/7a28283c-3f57-4276-a7d1-a7792377f37b-kube-api-access-8gbk5\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.353573 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.353548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:46.369318 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.369460 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.369460 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.369460 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.369608 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.369608 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ljf\" (UniqueName: \"kubernetes.io/projected/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-api-access-d6ljf\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.370078 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.369834 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.370237 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.370210 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.370392 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.370277 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.372494 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.372467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.372634 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.372614 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.427021 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.426988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ljf\" (UniqueName: \"kubernetes.io/projected/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-api-access-d6ljf\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.457900 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.457805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:46.465135 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.465109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:46.469887 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:46.469857 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a28283c_3f57_4276_a7d1_a7792377f37b.slice/crio-6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83 WatchSource:0}: Error finding container 6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83: Status 404 returned error can't find the container with id 6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83 Nov 01 08:32:46.633586 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.633552 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:46.671458 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.671418 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p7p9g" event={"ID":"7a28283c-3f57-4276-a7d1-a7792377f37b","Type":"ContainerStarted","Data":"6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83"} Nov 01 08:32:46.673230 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.673190 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fbd6bf9f-hrn97" event={"ID":"af4e88fb-f709-4c95-828d-9ef46ff8e159","Type":"ContainerStarted","Data":"d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6"} Nov 01 08:32:46.673230 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.673236 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fbd6bf9f-hrn97" event={"ID":"af4e88fb-f709-4c95-828d-9ef46ff8e159","Type":"ContainerStarted","Data":"3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871"} Nov 01 08:32:46.730488 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.730408 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5"] Nov 01 08:32:46.737967 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:46.737931 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee31b8bb_4ebe_41dd_a701_66eaf42137da.slice/crio-47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75 WatchSource:0}: Error finding container 47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75: Status 404 returned error can't find the container with id 47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75 Nov 01 08:32:46.848625 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.848594 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc"] Nov 01 08:32:46.853857 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:46.853807 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66fbd6bf9f-hrn97" podStartSLOduration=1.853793209 podStartE2EDuration="1.853793209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:46.853194782 +0000 UTC m=+63.942701070" watchObservedRunningTime="2026-11-01 08:32:46.853793209 +0000 UTC m=+63.943299497" Nov 01 08:32:46.857965 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:46.857931 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd398bcb3_f7fc_4307_bbc3_6acccfcdc43c.slice/crio-b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b WatchSource:0}: Error finding container b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b: Status 404 returned error can't find the container with id b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b Nov 01 08:32:47.677984 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:47.677947 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" event={"ID":"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c","Type":"ContainerStarted","Data":"b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b"} Nov 01 08:32:47.680579 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:47.680540 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" event={"ID":"ee31b8bb-4ebe-41dd-a701-66eaf42137da","Type":"ContainerStarted","Data":"f77c393411c7c14816a095a3a3e9b4be26b463278ba6f26a9c40ed5a9c92f93c"} Nov 01 08:32:47.680765 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:47.680604 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" event={"ID":"ee31b8bb-4ebe-41dd-a701-66eaf42137da","Type":"ContainerStarted","Data":"9a0bd6df2763914c9f70d7417deaf7f851de796efe0422093023e3202fa88661"} Nov 01 08:32:47.680765 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:47.680624 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" event={"ID":"ee31b8bb-4ebe-41dd-a701-66eaf42137da","Type":"ContainerStarted","Data":"47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75"} Nov 01 08:32:48.289137 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.289104 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:32:48.293052 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.293025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.343385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343245 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 08:32:48.343385 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343299 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 08:32:48.343653 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343454 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 08:32:48.343653 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343542 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7q4h\"" Nov 01 08:32:48.343653 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343577 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 08:32:48.343887 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343660 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 08:32:48.343887 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343685 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 08:32:48.344014 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.343999 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 08:32:48.349234 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.349206 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 08:32:48.387695 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.387658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.387695 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.387696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.387931 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.387729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.387931 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.387799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.387931 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.387866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.387931 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.387907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388048 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388048 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388104 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388070 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388140 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388169 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388203 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.388259 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.388245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.436424 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.436385 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 08:32:48.438005 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.437981 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:32:48.489078 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.488975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489078 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489310 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489310 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489310 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489310 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489310 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489310 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489600 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489600 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489600 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489600 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.489600 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.489452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.490373 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.490060 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.490373 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.490132 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.492882 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.492517 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.492882 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.492678 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.492882 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.492738 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.493252 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.493223 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.493518 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.493494 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.494087 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.494041 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.494361 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.494340 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.494689 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.494668 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.495499 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.495477 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.495597 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.495579 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.550643 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.550607 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.620276 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.620234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:48.693451 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.693412 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a28283c-3f57-4276-a7d1-a7792377f37b" containerID="d5419398af8a5c58dfc0c77f86bbc8f9fa9be4b2db018a9fb4adad4ffac78a8a" exitCode=0 Nov 01 08:32:48.693827 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:48.693489 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p7p9g" event={"ID":"7a28283c-3f57-4276-a7d1-a7792377f37b","Type":"ContainerDied","Data":"d5419398af8a5c58dfc0c77f86bbc8f9fa9be4b2db018a9fb4adad4ffac78a8a"} Nov 01 08:32:49.051046 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.050955 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:32:49.059066 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:49.059030 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd38f917_f55b_458f_bfd4_18ea32a077af.slice/crio-652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf WatchSource:0}: Error finding container 652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf: Status 404 returned error can't find the container with id 652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf Nov 01 08:32:49.196554 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.196508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:49.245122 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.245090 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 08:32:49.249839 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.249807 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:49.297381 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.297338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:49.345603 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.345578 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 08:32:49.350295 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.350263 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:49.398848 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.398811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:49.446971 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.446949 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 08:32:49.454636 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.454600 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\"" Nov 01 08:32:49.457621 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.457600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:49.467198 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.467182 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\"" Nov 01 08:32:49.469203 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.469186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:49.499752 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.497956 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 08:32:49.504019 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.503993 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:49.699570 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.699478 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" event={"ID":"ee31b8bb-4ebe-41dd-a701-66eaf42137da","Type":"ContainerStarted","Data":"aa98842208e43078a8db594fffab7761e861beab57ddf23ab12b3f622912c120"} Nov 01 08:32:49.701117 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.701091 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf"} Nov 01 08:32:49.703561 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.703533 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p7p9g" event={"ID":"7a28283c-3f57-4276-a7d1-a7792377f37b","Type":"ContainerStarted","Data":"7872dcd1845d7e3538461bf60419b5f2c034ecc812f28a9b1933fd87bce812de"} Nov 01 08:32:49.703672 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.703586 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p7p9g" event={"ID":"7a28283c-3f57-4276-a7d1-a7792377f37b","Type":"ContainerStarted","Data":"08c1067ec9bb41854e96b9fd0c4e74705f557c28c997c7685902638747cf0a2e"} Nov 01 08:32:49.729208 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.729180 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\"" Nov 01 08:32:49.732191 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.732167 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:49.810254 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.810207 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" podStartSLOduration=2.262540549 podStartE2EDuration="4.810191453s" totalImagesPullingTime="2.547650904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:49.809871923 +0000 UTC m=+66.899378212" watchObservedRunningTime="2026-11-01 08:32:49.810191453 +0000 UTC m=+66.899697719" Nov 01 08:32:49.835515 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.835481 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gxwrb"] Nov 01 08:32:49.844148 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:49.844112 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d57cfa0_eef3_450d_9030_3049ee009711.slice/crio-9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954 WatchSource:0}: Error finding container 9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954: Status 404 returned error can't find the container with id 9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954 Nov 01 08:32:49.885869 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.885836 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tg7cb"] Nov 01 08:32:49.894094 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:49.894065 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc68d9cc_01a2_49b3_b0f6_f510d04caefa.slice/crio-f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c WatchSource:0}: Error finding container f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c: Status 404 returned error can't find the container with id f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c Nov 01 08:32:49.922234 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:49.922176 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p7p9g" podStartSLOduration=2.117715985 podStartE2EDuration="3.922161259s" totalImagesPullingTime="1.722554193s" totalInitContainerRuntime="81.891081ms" isStatefulPod=true podCreationTimestamp="2026-11-01 08:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:49.921132691 +0000 UTC m=+67.010638979" watchObservedRunningTime="2026-11-01 08:32:49.922161259 +0000 UTC m=+67.011667547" Nov 01 08:32:50.106086 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.106054 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7b55"] Nov 01 08:32:50.114339 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:50.114306 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1dcebb_bbfe_4e4a_b684_84012ddc57b8.slice/crio-2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49 WatchSource:0}: Error finding container 2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49: Status 404 returned error can't find the container with id 2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49 Nov 01 08:32:50.708753 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.708703 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7b55" event={"ID":"af1dcebb-bbfe-4e4a-b684-84012ddc57b8","Type":"ContainerStarted","Data":"2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49"} Nov 01 08:32:50.710381 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.710349 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gxwrb" event={"ID":"7d57cfa0-eef3-450d-9030-3049ee009711","Type":"ContainerStarted","Data":"9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954"} Nov 01 08:32:50.711968 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.711938 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tg7cb" event={"ID":"fc68d9cc-01a2-49b3-b0f6-f510d04caefa","Type":"ContainerStarted","Data":"f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c"} Nov 01 08:32:50.859314 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.859276 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn"] Nov 01 08:32:50.863757 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.863735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:50.911685 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.911659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e698209-ef38-47d5-be80-4e5208813720-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:50.911846 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.911702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526sp\" (UniqueName: \"kubernetes.io/projected/0e698209-ef38-47d5-be80-4e5208813720-kube-api-access-526sp\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:50.917802 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.917750 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 08:32:50.917802 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.917788 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h6qxd\"" Nov 01 08:32:50.919745 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:50.919702 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn"] Nov 01 08:32:51.012864 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.012828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e698209-ef38-47d5-be80-4e5208813720-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:51.013057 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.012882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526sp\" (UniqueName: \"kubernetes.io/projected/0e698209-ef38-47d5-be80-4e5208813720-kube-api-access-526sp\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:51.015668 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.015636 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e698209-ef38-47d5-be80-4e5208813720-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:51.071302 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.071272 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526sp\" (UniqueName: \"kubernetes.io/projected/0e698209-ef38-47d5-be80-4e5208813720-kube-api-access-526sp\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:51.191958 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.191863 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:51.583195 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.583109 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn"] Nov 01 08:32:51.606936 ip-10-0-143-16 kubenswrapper[2580]: W1101 08:32:51.606881 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e698209_ef38_47d5_be80_4e5208813720.slice/crio-8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c WatchSource:0}: Error finding container 8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c: Status 404 returned error can't find the container with id 8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c Nov 01 08:32:51.734788 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.733774 2580 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" exitCode=0 Nov 01 08:32:51.734788 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.733859 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} Nov 01 08:32:51.737340 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.737304 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" event={"ID":"0e698209-ef38-47d5-be80-4e5208813720","Type":"ContainerStarted","Data":"8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c"} Nov 01 08:32:51.740614 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.740441 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" event={"ID":"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c","Type":"ContainerStarted","Data":"dd055bd91276fed89b27092f3dfb4cc21e0729c40b192d6dbdd3bcee80c16370"} Nov 01 08:32:51.740614 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.740545 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" event={"ID":"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c","Type":"ContainerStarted","Data":"31b62ec77b65c0c85457efaa872370c0bed0d3620522688507924283d5dd96c9"} Nov 01 08:32:51.740614 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:51.740577 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" event={"ID":"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c","Type":"ContainerStarted","Data":"214a94fca5777f806d2251c8411a9669ece634b2f290422edd75b35fde9938c3"} Nov 01 08:32:52.749751 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:52.749696 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gxwrb" event={"ID":"7d57cfa0-eef3-450d-9030-3049ee009711","Type":"ContainerStarted","Data":"9fa877b8fd938865518bf652a6f109c94f3e298c8fdb2e2e1a2ce9025e929d3d"} Nov 01 08:32:52.750222 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:52.749779 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gxwrb" event={"ID":"7d57cfa0-eef3-450d-9030-3049ee009711","Type":"ContainerStarted","Data":"2fdd679023062e8cd87550fe1d4ae56e98b228f6c69032ba8b07d64fe7b38b28"} Nov 01 08:32:52.860359 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:52.860288 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" podStartSLOduration=3.07520564 podStartE2EDuration="6.860264457s" totalImagesPullingTime="3.785058817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:51.973954511 +0000 UTC m=+69.063460800" watchObservedRunningTime="2026-11-01 08:32:52.860264457 +0000 UTC m=+69.949770746" Nov 01 08:32:53.476616 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.474920 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gxwrb" podStartSLOduration=67.310525105 podStartE2EDuration="1m9.474905696s" totalImagesPullingTime="2.164380591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:52.860907703 +0000 UTC m=+69.950413992" watchObservedRunningTime="2026-11-01 08:32:53.474905696 +0000 UTC m=+70.564411983" Nov 01 08:32:53.476616 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.476276 2580 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fbd6bf9f-hrn97"] Nov 01 08:32:53.513639 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.513550 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6958777449-f4bt5"] Nov 01 08:32:53.521005 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.520962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.591484 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.591443 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6958777449-f4bt5"] Nov 01 08:32:53.654348 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654541 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654541 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654541 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654541 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654541 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654541 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.654851 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.654589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755089 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755089 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755631 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.755972 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.755921 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.756084 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.756058 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.757020 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.756961 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.757138 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.757026 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.757337 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.757288 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.759342 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.759296 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.760393 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.760346 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.816601 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.816573 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:53.840682 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:53.840431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:54.287171 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.287135 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6958777449-f4bt5"] Nov 01 08:32:54.763775 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.763508 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} Nov 01 08:32:54.763775 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.763603 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} Nov 01 08:32:54.763775 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.763645 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} Nov 01 08:32:54.763775 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.763685 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} Nov 01 08:32:54.763775 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.763756 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} Nov 01 08:32:54.766751 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.766695 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" event={"ID":"0e698209-ef38-47d5-be80-4e5208813720","Type":"ContainerStarted","Data":"53f3197ca95aa6a18b19d4177a5094f7d9e0ac819b80e24056e7d648b8e5fb86"} Nov 01 08:32:54.767121 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.767100 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:54.769438 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.769408 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6958777449-f4bt5" event={"ID":"17bada36-7447-4f33-9279-ef2b7c1a0c33","Type":"ContainerStarted","Data":"656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b"} Nov 01 08:32:54.769630 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.769458 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6958777449-f4bt5" event={"ID":"17bada36-7447-4f33-9279-ef2b7c1a0c33","Type":"ContainerStarted","Data":"96d3c09554f8bbc57cc70a67593ebe91a64a34bbaa89c4bb2210a20eed65e473"} Nov 01 08:32:54.773357 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.773329 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:54.985741 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:54.985663 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" podStartSLOduration=2.717066335 podStartE2EDuration="4.985648391s" totalImagesPullingTime="2.268582056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:54.876651443 +0000 UTC m=+71.966157732" watchObservedRunningTime="2026-11-01 08:32:54.985648391 +0000 UTC m=+72.075154678" Nov 01 08:32:55.097815 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:55.097745 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6958777449-f4bt5" podStartSLOduration=2.097706626 podStartE2EDuration="2.097706626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:55.09642775 +0000 UTC m=+72.185934039" watchObservedRunningTime="2026-11-01 08:32:55.097706626 +0000 UTC m=+72.187212914" Nov 01 08:32:55.776395 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:55.776357 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7b55" event={"ID":"af1dcebb-bbfe-4e4a-b684-84012ddc57b8","Type":"ContainerStarted","Data":"c346eb59edef6d427d170cb1f466512dce4ab501c1abed9a0b274ac4da61287a"} Nov 01 08:32:55.885881 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:55.885807 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7b55" podStartSLOduration=66.721279365 podStartE2EDuration="1m11.885787232s" totalImagesPullingTime="5.164507867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:32:55.885241195 +0000 UTC m=+72.974747484" watchObservedRunningTime="2026-11-01 08:32:55.885787232 +0000 UTC m=+72.975293521" Nov 01 08:32:55.930218 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:55.930185 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:56.784170 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:56.784129 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerStarted","Data":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} Nov 01 08:32:56.785993 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:56.785952 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tg7cb" event={"ID":"fc68d9cc-01a2-49b3-b0f6-f510d04caefa","Type":"ContainerStarted","Data":"80ed475ceb3d5317b484cea2c7b51cb13e3d7081690fcac8ffea04cfe5be62de"} Nov 01 08:32:56.786345 ip-10-0-143-16 kubenswrapper[2580]: I1101 08:32:56.786321 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:56.829508 ip-10-0-143-16 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 08:32:56.838489 ip-10-0-143-16 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 08:32:56.838673 ip-10-0-143-16 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 08:32:56.838743 ip-10-0-143-16 systemd[1]: kubelet.service: Consumed 4.625s CPU time. Nov 01 08:32:56.854630 ip-10-0-143-16 systemd[1]: Starting Kubernetes Kubelet... Nov 01 08:32:56.948442 ip-10-0-143-16 kubenswrapper[5710]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 08:32:56.948442 ip-10-0-143-16 kubenswrapper[5710]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 08:32:56.948442 ip-10-0-143-16 kubenswrapper[5710]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 08:32:56.948442 ip-10-0-143-16 kubenswrapper[5710]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 08:32:56.950822 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950676 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.950822 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950819 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.950822 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950825 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.950822 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950830 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950834 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950839 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950843 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950848 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950852 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950870 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950874 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950878 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950882 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950886 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950890 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950894 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950897 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950902 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950906 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950910 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950914 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950917 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.951126 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950922 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950926 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950931 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950934 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950938 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950944 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950948 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950952 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950956 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950960 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950963 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950967 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950971 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950977 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950983 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950987 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950992 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.950997 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951001 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.951890 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951006 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951011 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951016 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951020 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951024 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951028 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951032 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951035 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951040 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951045 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951049 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951053 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951057 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951061 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951064 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951069 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951073 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951077 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951081 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951086 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.952600 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951090 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951094 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951098 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951102 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951106 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951110 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951114 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951117 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951121 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951125 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951129 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951133 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951137 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951141 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951146 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951150 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951155 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951159 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951163 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.953102 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951167 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951171 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951175 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951179 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951183 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951187 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951190 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951194 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951198 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951202 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951206 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951212 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951216 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951221 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951230 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951234 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951247 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951251 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951255 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951260 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.953615 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951263 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951268 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951272 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951276 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951280 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.951284 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952491 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952501 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952511 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952515 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952518 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952520 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952523 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952526 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952528 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952531 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952534 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952538 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952540 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952543 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.954264 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952545 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952548 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952550 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952553 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952556 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952558 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952561 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952564 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952567 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952569 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952572 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952574 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952577 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952579 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952582 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952585 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952588 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952590 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952593 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.954900 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952596 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952598 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952606 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952609 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952612 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952615 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952617 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952620 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952623 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952625 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952628 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952631 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952633 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952636 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952639 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952641 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952643 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952646 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952649 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952651 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.955381 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952654 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952657 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952659 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952662 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952665 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952668 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952670 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952673 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952676 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952678 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952681 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952685 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952690 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952693 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952696 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952699 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952702 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952705 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952709 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.955891 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952727 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952731 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952734 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952737 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952739 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952742 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952744 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952747 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952750 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952753 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952755 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952758 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952761 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952764 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952766 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952769 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952772 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952774 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952777 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952779 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.956340 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952782 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952785 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952810 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952814 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952817 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952820 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952823 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952826 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952828 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952831 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952834 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952838 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952841 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.952844 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953431 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953440 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953445 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953448 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953452 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953456 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.956879 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953460 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953465 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953470 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953474 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953479 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953482 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953486 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953490 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953493 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953498 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953501 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953506 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953510 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953515 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953519 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953524 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953528 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953532 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953536 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953540 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.957355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953545 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953550 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953553 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953557 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953561 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953568 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953573 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953577 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953580 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953584 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953588 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953592 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953596 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953600 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953606 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953612 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953617 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953622 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953627 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953631 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.957884 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953635 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953640 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953644 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953649 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953653 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953658 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953661 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953663 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953666 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953668 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953671 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953674 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953677 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953679 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953683 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953686 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953689 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953694 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953696 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.958378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953699 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953702 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953704 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953707 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953710 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953729 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953732 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953735 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953738 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953740 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953743 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953745 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953748 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953751 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953753 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953756 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953759 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953761 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953764 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.958886 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953766 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953772 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953775 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953778 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953781 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953784 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953787 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953789 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953792 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953795 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953797 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953802 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953804 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953807 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953809 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953812 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953814 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953817 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953819 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.959355 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953822 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953824 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.953827 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.953970 5710 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.953984 5710 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.953991 5710 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.953997 5710 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954002 5710 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954005 5710 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954010 5710 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954018 5710 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954022 5710 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954025 5710 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954029 5710 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954032 5710 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954036 5710 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954039 5710 flags.go:64] FLAG: --cgroup-root="" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954042 5710 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954045 5710 flags.go:64] FLAG: --client-ca-file="" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954047 5710 flags.go:64] FLAG: --cloud-provider="external" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954050 5710 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954056 5710 flags.go:64] FLAG: --cluster-domain="" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954059 5710 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 08:32:56.959842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954062 5710 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954069 5710 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954073 5710 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954077 5710 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954081 5710 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954085 5710 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954088 5710 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954091 5710 flags.go:64] FLAG: --contention-profiling="false" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954094 5710 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954098 5710 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954101 5710 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954104 5710 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954109 5710 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954112 5710 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954115 5710 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954118 5710 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954121 5710 flags.go:64] FLAG: --enable-server="true" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954124 5710 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954128 5710 flags.go:64] FLAG: --event-burst="100" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954131 5710 flags.go:64] FLAG: --event-qps="50" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954134 5710 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954137 5710 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954140 5710 flags.go:64] FLAG: --eviction-hard="" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954145 5710 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 08:32:56.960538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954148 5710 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954152 5710 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954155 5710 flags.go:64] FLAG: --eviction-soft="" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954158 5710 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954161 5710 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954164 5710 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954167 5710 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954170 5710 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954173 5710 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954179 5710 flags.go:64] FLAG: --feature-gates="" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954184 5710 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954187 5710 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954191 5710 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954194 5710 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954197 5710 flags.go:64] FLAG: --healthz-port="10248" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954200 5710 flags.go:64] FLAG: --help="false" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954204 5710 flags.go:64] FLAG: --hostname-override="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954207 5710 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954210 5710 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954213 5710 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954217 5710 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954220 5710 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954223 5710 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 08:32:56.961483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954226 5710 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954229 5710 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954232 5710 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954235 5710 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954239 5710 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954242 5710 flags.go:64] FLAG: --kube-reserved="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954245 5710 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954248 5710 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954251 5710 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954274 5710 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954278 5710 flags.go:64] FLAG: --lock-file="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954281 5710 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954285 5710 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954288 5710 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954294 5710 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954297 5710 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954300 5710 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954304 5710 flags.go:64] FLAG: --logging-format="text" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954309 5710 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954314 5710 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954317 5710 flags.go:64] FLAG: --manifest-url="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954320 5710 flags.go:64] FLAG: --manifest-url-header="" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954325 5710 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954328 5710 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954333 5710 flags.go:64] FLAG: --max-pods="110" Nov 01 08:32:56.962154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954336 5710 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954339 5710 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954342 5710 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954345 5710 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954348 5710 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954351 5710 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954354 5710 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954362 5710 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954365 5710 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954368 5710 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954372 5710 flags.go:64] FLAG: --pod-cidr="" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954375 5710 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954378 5710 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954381 5710 flags.go:64] FLAG: --pods-per-core="0" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954385 5710 flags.go:64] FLAG: --port="10250" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954388 5710 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954390 5710 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0506531ba9bd3669b" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954394 5710 flags.go:64] FLAG: --qos-reserved="" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954397 5710 flags.go:64] FLAG: --read-only-port="10255" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954399 5710 flags.go:64] FLAG: --register-node="true" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954402 5710 flags.go:64] FLAG: --register-with-taints="" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954407 5710 flags.go:64] FLAG: --registry-burst="10" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954410 5710 flags.go:64] FLAG: --registry-qps="5" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954413 5710 flags.go:64] FLAG: --reserved-cpus="" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954416 5710 flags.go:64] FLAG: --reserved-memory="" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954421 5710 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 08:32:56.962878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954424 5710 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954428 5710 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954431 5710 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954434 5710 flags.go:64] FLAG: --runonce="false" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954437 5710 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954440 5710 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954443 5710 flags.go:64] FLAG: --seccomp-default="false" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954446 5710 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954449 5710 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954452 5710 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954456 5710 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954459 5710 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954462 5710 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954465 5710 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954470 5710 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954475 5710 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954479 5710 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954485 5710 flags.go:64] FLAG: --system-cgroups="" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954490 5710 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954497 5710 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954501 5710 flags.go:64] FLAG: --tls-cert-file="" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954505 5710 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954510 5710 flags.go:64] FLAG: --tls-min-version="" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954514 5710 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954519 5710 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 08:32:56.963500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954523 5710 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954528 5710 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954532 5710 flags.go:64] FLAG: --v="2" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954539 5710 flags.go:64] FLAG: --version="false" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954546 5710 flags.go:64] FLAG: --vmodule="" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954552 5710 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.954559 5710 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954765 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954771 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954776 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954779 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954782 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954785 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954787 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954790 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954792 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954795 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954797 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954800 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954802 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954805 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.964245 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954808 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954810 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954813 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954815 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954818 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954820 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954825 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954828 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954830 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954832 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954835 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954837 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954840 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954843 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954845 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954847 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954850 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954854 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954856 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954859 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.965035 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954861 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954864 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954867 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954869 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954872 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954875 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954877 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954880 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954882 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954886 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954890 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954893 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954895 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954898 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954901 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954903 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954906 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954909 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954913 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.965796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954915 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954918 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954921 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954924 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954926 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954929 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954931 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954934 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954936 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954939 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954943 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954945 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954947 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954950 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954952 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954955 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954958 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954960 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954965 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954968 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.966492 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954971 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954973 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954976 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954979 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954982 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954985 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954987 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954990 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954992 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954995 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.954997 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955001 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955004 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955006 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955009 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955011 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955014 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955016 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955019 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.967159 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955021 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955024 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955027 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955031 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955034 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955036 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955039 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955041 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955044 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955047 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955050 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955052 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955055 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955057 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.955088 5710 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955124 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.967873 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955128 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955131 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955133 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955136 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955139 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955141 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955144 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955146 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955149 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955153 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955155 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955158 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955161 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955164 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955166 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955169 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955171 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955174 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955176 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.968477 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955181 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955184 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955186 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955189 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955191 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955194 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955197 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955199 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955203 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955205 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955207 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955210 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955212 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955215 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955217 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955220 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955222 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955225 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955227 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955230 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.969254 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955232 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955235 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955248 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955251 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955253 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955256 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955259 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955261 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955264 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955266 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955269 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955271 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955276 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955278 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955280 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955283 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955285 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955290 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955293 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.970224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955296 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955299 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955302 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955305 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955307 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955310 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955312 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955315 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955317 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955320 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955323 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955325 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955327 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955330 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955332 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955336 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955338 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955341 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955344 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955347 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.970927 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955350 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955352 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955355 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955357 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955360 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955363 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955368 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955379 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955381 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955384 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955387 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955390 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955393 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955395 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955398 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955401 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955404 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955407 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955409 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.971684 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955412 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955414 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955417 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955419 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955422 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955424 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955427 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.955430 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.959596 5710 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.959617 5710 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.959637 5710 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.959642 5710 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959703 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959708 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959711 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959729 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959733 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959735 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959739 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.972466 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959742 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959745 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959747 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959750 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959753 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959756 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959758 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959761 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959764 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959767 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959769 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959772 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959787 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959791 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959795 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959798 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959801 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959804 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959807 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.973188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959809 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959812 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959815 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959819 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959822 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959825 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959827 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959832 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959836 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959839 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959842 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959845 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959848 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959851 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959854 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959857 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959859 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959862 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959865 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.973801 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959867 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959870 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959873 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959875 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959878 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959881 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959883 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959886 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959889 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959891 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959894 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959896 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959899 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959901 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959904 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959907 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959909 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959912 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959915 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.974274 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959917 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959920 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959923 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959925 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959928 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959930 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959933 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959936 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959938 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959941 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959943 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959946 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959948 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959951 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959954 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959956 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959959 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959961 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959964 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959966 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.974754 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959969 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959971 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959973 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959976 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959979 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959981 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959984 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959986 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959988 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959991 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959993 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959996 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.959999 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960001 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960005 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960008 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960011 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960013 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960016 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960019 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.975228 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960021 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.960024 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.960053 5710 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961115 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961132 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961137 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961141 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961146 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961150 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961154 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961159 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961163 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961167 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961177 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961181 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.975837 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961185 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961190 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961194 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961199 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961204 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961208 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961212 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961216 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961220 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961252 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961256 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961266 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961270 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961274 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961278 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961283 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961288 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961292 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961297 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961301 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.976215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961305 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961310 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961314 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961323 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961327 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961331 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961336 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961340 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961344 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961349 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961353 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961357 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961361 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961365 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961369 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961505 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961516 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961520 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961524 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.976692 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961528 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961533 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961537 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961574 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961579 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961583 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961587 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961591 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961595 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961606 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961610 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961614 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961621 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961625 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961630 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961634 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961638 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961640 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961643 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961646 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.977205 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961649 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961651 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961654 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961659 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961667 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961670 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961673 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961676 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961679 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961684 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961688 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961692 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961696 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961701 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961705 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961711 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961806 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961969 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961976 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.977678 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961980 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961983 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961987 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961989 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961992 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961994 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.961997 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962000 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962002 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962005 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962007 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962010 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962012 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962014 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962017 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962020 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.962058 5710 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 08:32:56.978292 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962106 5710 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962112 5710 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962115 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962118 5710 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962120 5710 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962123 5710 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962126 5710 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962128 5710 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962131 5710 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962137 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962141 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962143 5710 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962146 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962149 5710 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962152 5710 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962156 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962158 5710 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962161 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962164 5710 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962167 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 08:32:56.979062 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962169 5710 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962173 5710 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962175 5710 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962178 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962182 5710 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962186 5710 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962189 5710 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962192 5710 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962195 5710 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962198 5710 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962200 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962203 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962205 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962208 5710 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962210 5710 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962212 5710 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962216 5710 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962220 5710 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962223 5710 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 08:32:56.979829 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962227 5710 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962230 5710 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962233 5710 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962235 5710 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962238 5710 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962240 5710 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962243 5710 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962246 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962250 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962253 5710 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962256 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962258 5710 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962261 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962263 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962265 5710 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962268 5710 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962270 5710 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962273 5710 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962275 5710 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962277 5710 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 08:32:56.980546 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962280 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962283 5710 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962285 5710 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962288 5710 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962291 5710 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962293 5710 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962296 5710 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962299 5710 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962301 5710 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962303 5710 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962306 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962309 5710 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962311 5710 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962314 5710 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962317 5710 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962319 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962323 5710 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962325 5710 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962328 5710 feature_gate.go:386] unrecognized feature gate: Example Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962330 5710 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 08:32:56.981455 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962334 5710 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962337 5710 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962339 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962342 5710 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962344 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962348 5710 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962351 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962353 5710 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962356 5710 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962358 5710 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962361 5710 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962363 5710 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962366 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962368 5710 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962371 5710 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962374 5710 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962376 5710 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962379 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962382 5710 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 08:32:56.982339 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962384 5710 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962387 5710 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962390 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962394 5710 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962397 5710 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962399 5710 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962402 5710 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:32:56.962405 5710 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.962498 5710 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.963541 5710 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.963616 5710 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.963813 5710 server.go:1014] "Starting client certificate rotation" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.964926 5710 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.963977 5710 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.966645 5710 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 08:32:56.983212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.966756 5710 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 08:26:44 +0000 UTC" deadline="2028-08-08 02:13:01.074281674 +0000 UTC" Nov 01 08:32:56.983868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.966823 5710 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15497h40m4.10746242s" Nov 01 08:32:56.983868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.968110 5710 log.go:25] "Validated CRI v1 runtime API" Nov 01 08:32:56.983868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.970819 5710 log.go:25] "Validated CRI v1 image API" Nov 01 08:32:56.983868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.973757 5710 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 08:32:56.983868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.976205 5710 fs.go:135] Filesystem UUIDs: map[1ca88dd6-c6c4-4015-82fb-d3254545b3f7:/dev/nvme0n1p4 4fdad4ec-7f19-4642-afe3-aa58c6c757e7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 01 08:32:56.984139 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.976228 5710 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c/userdata/shm major:0 minor:399 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96d3c09554f8bbc57cc70a67593ebe91a64a34bbaa89c4bb2210a20eed65e473/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96d3c09554f8bbc57cc70a67593ebe91a64a34bbaa89c4bb2210a20eed65e473/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~projected/kube-api-access-fcssx:{mountpoint:/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~projected/kube-api-access-fcssx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e698209-ef38-47d5-be80-4e5208813720/volumes/kubernetes.io~projected/kube-api-access-526sp:{mountpoint:/var/lib/kubelet/pods/0e698209-ef38-47d5-be80-4e5208813720/volumes/kubernetes.io~projected/kube-api-access-526sp major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e698209-ef38-47d5-be80-4e5208813720/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/0e698209-ef38-47d5-be80-4e5208813720/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15ee2bde-64f4-4580-8969-c395abf4c2f0/volumes/kubernetes.io~projected/kube-api-access-kb8nb:{mountpoint:/var/lib/kubelet/pods/15ee2bde-64f4-4580-8969-c395abf4c2f0/volumes/kubernetes.io~projected/kube-api-access-kb8nb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15ee2bde-64f4-4580-8969-c395abf4c2f0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/15ee2bde-64f4-4580-8969-c395abf4c2f0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f1c6a3-98c1-4970-9476-3330567de96c/volumes/kubernetes.io~projected/kube-api-access-fld4z:{mountpoint:/var/lib/kubelet/pods/16f1c6a3-98c1-4970-9476-3330567de96c/volumes/kubernetes.io~projected/kube-api-access-fld4z major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~projected/kube-api-access-66p5s:{mountpoint:/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~projected/kube-api-access-66p5s major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~projected/kube-api-access-mnx4n:{mountpoint:/var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~projected/kube-api-access-mnx4n major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29716515-5846-432e-b2a2-e63ce1ea7059/volumes/kubernetes.io~projected/kube-api-access-ppxql:{mountpoint:/var/lib/kubelet/pods/29716515-5846-432e-b2a2-e63ce1ea7059/volumes/kubernetes.io~projected/kube-api-access-ppxql major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29716515-5846-432e-b2a2-e63ce1ea7059/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/29716515-5846-432e-b2a2-e63ce1ea7059/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/388ff44e-1500-4858-87d0-7a33eee493d3/volumes/kubernetes.io~projected/kube-api-access-5k7rj:{mountpoint:/var/lib/kubelet/pods/388ff44e-1500-4858-87d0-7a33eee493d3/volumes/kubernetes.io~projected/kube-api-access-5k7rj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volumes/kubernetes.io~projected/kube-api-access-mln9l:{mountpoint:/var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volumes/kubernetes.io~projected/kube-api-access-mln9l major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59c8058b-626b-4462-9765-5157b802d928/volumes/kubernetes.io~projected/kube-api-access-6j4fh:{mountpoint:/var/lib/kubelet/pods/59c8058b-626b-4462-9765-5157b802d928/volumes/kubernetes.io~projected/kube-api-access-6j4fh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6611ffa7-eb5a-4a4f-be97-2d444e688e8d/volumes/kubernetes.io~projected/kube-api-access-4b8mx:{mountpoint:/var/lib/kubelet/pods/6611ffa7-eb5a-4a4f-be97-2d444e688e8d/volumes/kubernetes.io~projected/kube-api-access-4b8mx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6611ffa7-eb5a-4a4f-be97-2d444e688e8d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6611ffa7-eb5a-4a4f-be97-2d444e688e8d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~projected/kube-api-access-8gbk5:{mountpoint:/var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~projected/kube-api-access-8gbk5 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d57cfa0-eef3-450d-9030-3049ee009711/volumes/kubernetes.io~projected/kube-api-access-76m8h:{mountpoint:/var/lib/kubelet/pods/7d57cfa0-eef3-450d-9030-3049ee009711/volumes/kubernetes.io~projected/kube-api-access-76m8h major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d57cfa0-eef3-450d-9030-3049ee009711/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7d57cfa0-eef3-450d-9030-3049ee009711/volumes/kubernetes.io~secret/metrics-certs major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/958489ac-f598-4365-a166-22371ca307d4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/958489ac-f598-4365-a166-22371ca307d4/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~projected/kube-api-access-cmhh4:{mountpoint:/var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~projected/kube-api-access-cmhh4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07fa048-d1fa-4b73-8120-6d04c02c91e9/volumes/kubernetes.io~projected/kube-api-access-cn7gx:{mountpoint:/var/lib/kubelet/pods/a07fa048-d1fa-4b73-8120-6d04c02c91e9/volumes/kubernetes.io~projected/kube-api-access-cn7gx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af1dcebb-bbfe-4e4a-b684-84012ddc57b8/volumes/kubernetes.io~projected/kube-api-access-rd87k:{mountpoint:/var/lib/kubelet/pods/af1dcebb-bbfe-4e4a-b684-84012ddc57b8/volumes/kubernetes.io~projected/kube-api-access-rd87k major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~projected/kube-api-access-7vf7g:{mountpoint:/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~projected/kube-api-access-7vf7g major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb599c9a-0381-45c2-801d-ef6578d0ea9d/volumes/kubernetes.io~projected/kube-api-access-qzggv:{mountpoint:/var/lib/kubelet/pods/bb599c9a-0381-45c2-801d-ef6578d0ea9d/volumes/kubernetes.io~projected/kube-api-access-qzggv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0c527af-be39-4887-990a-0dd793e2a751/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/d0c527af-be39-4887-990a-0dd793e2a751/volumes/kubernetes.io~secret/tls-certificates major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~projected/kube-api-access-d6ljf:{mountpoint:/var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~projected/kube-api-access-d6ljf major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/alertmanager/9 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/config-reloader/6 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~empty-dir/config-out major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~projected/kube-api-access-zx9bq:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~projected/kube-api-access-zx9bq major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~projected/tls-assets major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/config-volume major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/web-config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e97a3a69-d419-495c-877f-651eae6172f7/volumes/kubernetes.io~projected/kube-api-access-57grl:{mountpoint:/var/lib/kubelet/pods/e97a3a69-d419-495c-877f-651eae6172f7/volumes/kubernetes.io~projected/kube-api-access-57grl major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~projected/kube-api-access-cx75b:{mountpoint:/var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~projected/kube-api-access-cx75b major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc68d9cc-01a2-49b3-b0f6-f510d04caefa/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fc68d9cc-01a2-49b3-b0f6-f510d04caefa/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:361 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c251531efec31e549359605bf80e891b4dd52f8343eb1b96bf580ccb09b24cd1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cfdb800697a657fc14d82d362efb8a221a3ccfe7070a7a149ffba4ace9444604/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1da7649df4dfc01bc6d73181c0d8575d654e0bc5fd1b3448c004c44020168485/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/dc6d8bd2969087b9967a193f14059a9c39fcdecadc8485e3fd3cd3a8d5033a9f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/3488c414bb1ff20d071ff3cb6a06b63d427a864b1ae6cef97c21ff3c2e7a7233/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/05c45f39923876fac435385f7beb89ad3472a97a22f92d72d7473ff18a871761/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2e2d85186dd8ef43ffa770272e5434a253c5be7759ac38444c5bdd7dbe33ad30/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0b90d9d005bf20de8a2aa318bf44e61a268781ca275f7a15a8f7f4a4d07c43e0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5678286922d31085ae17eded7253a458ca56ab45a7ae00da1136d1bd585fca69/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/89ce2dc73f74b9dda06949b4babca9e3048b74397b0a8020fb064e0275f06e64/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/cde35bc19295afd6dd7b145a0a3c2e332df1bdba561adcfe28800a4901187cb0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/6007d23309824cd9e63637848742f650adcfcd5999995f570d8c9b81aee1e6ed/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/ab3b0020b5aefbc3cc84beb052375e28cf0a76c134c0b0c2673d587c2bdcdceb/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/af97e6452e3591f7ecbc15327e3d10634129c780154570bd72d7d7902a99f42d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a35c7ae67990c0a7179484eb78c860fd7ce1e4a20a6b818b25fbfe1fc81af50d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/854199a57697945b935338de8a6f05ac39c24d937b5b7cb535749c08978b807f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/db4d6e0d655fa86908a310c4062b652f6e3cac6dc0028f4874695ad8b0e781a3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/354864df39e73c01f41c772e3db7dc95616a258250398e5b04d43c8062137e6c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2d7e530b770a2a9c966f80f20693c0ee1ab4c69f503fe635e426bd40f88284de/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/26270bba5e622005e0bedb2a9fae0e873bd1c1d792ecfc85e59981afe356a730/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c7f250e04c76466868dd033c8637ef8e9d639f995f510a0ddd66ead901ecd47a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/0646cbdf56aef292b78a1e5121969e8c8e527bd9c7f2b944bcf802465706ef84/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/7014e5d480209ed36d11c317618cbdc57f9211d2b982b18e2fce49076275015c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/81007ff38e40296d89cfd40bf770493066c11edd8ab51e4ea4d3a2e695e90dda/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/17e4c39036caffe07d36fde8417683e1ec2071a09e5db290275c9c621bb2434c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/9254bf82ba375d8f044860825d33e8cc738ea9e1825d3c0419a5070dbb4e2756/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/8b8229b36cbd1751a438af4d2af886a35fc6d2ad0e13527a2a33d68cc99c3847/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/34d22a0ebaed7bd01cfeed5ed5af2573b13e536692ef9d1e425e311db8700410/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/7ee7ea276b532bb99785c76f2533782a33175bc2a3dfc93a6dfdf42446fc6653/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/cf101b070b4ef557c0a6c6ad2b534d2e5cd320269c48ce6c603ff87a344531ca/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d944e0cd06bba22a80b69814e158f24514c318ccce97cceecb5123f2ad76e42b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/e627afaba22128f15bd012b09e2445c1ebbd2ed17b782e408cd78dc8804be696/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/9a923d2eaddf23dc5250dbd74ff5f0ce667430e0a8136e908e76ad01ba18b721/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/faa87762e7dc4b0739a4c8604863afc497ca6a9eabeee4dc5bde8dc740f7b97e/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/e27efcb40295d36d015d4508167fdd20fc495752447e722bb84cd9fa90cf518f/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/569fff88206a329dfd27597d32675b31e46c45ecc6d4e59113e5c65c0f14408d/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/35748567a9a5a6370be2822e8454a2248ec357d694f1c92f86d9bf9ee7a5cf69/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/099f6a921f7c02f5a4c5c93d69f1d46c7b77a7c5b13e455503787a7151071ab0/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/c4ab384fe3b0dd11ae1ac479cc0d0237a4456f240cef547898292d4934b3df21/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/3dabc5c09119b4db5974b0b475a95ac4c711b26d6b17626c983df01bb7274fb4/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/6736a8c29b11299442ac0192c4a55cb51e62843891210ec0a12396e03ddea56d/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/650fbfe4da116446293f8aa04bf195106e3396364cb6469d0f3f3a7ebff70d46/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/e759100700a03cbedcce1eeae3ce21ba9108414dfe7d93bfbecaaf755ae4dabb/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/cfbf89d1488e4cca38de5fe2e0a270d75a8e5a48fb1a157aef1e73e279fe1282/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/23687811e688c17983e1e4ee3844571067cb1c6af035ac15176fce3115087f20/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/578bbcb8ea0a77b048177d1a2b802bf0008adf0bbd86fdeafc45dea16dd3324a/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/c2a83e74f37527aaebb8fe19c1c8c0a17e6c9feb8e3ebffb0ae0d5dca48d1cc5/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/b94f25addd65125ab73ae7e3b79a5e9cad826bae2e9a70d5c57b7b70f4751312/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/1739e9a53a3e150efd92e735fc295da1cd05e6912a3c09d764b7cb5e25929d3e/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/66bfe685c8670de224744213f4ad90c79453968774553a52df82490242018964/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/2f5b2d4e3af532bbd87a322bdcc2b1bc0a9209bea0ad1aefcd94dad2ab3b6b65/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/1d9585e14bb1c7a7cb23a9b8ef7dac6a0f9c17c6db60847afd51e3f77c93bb23/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/d05301a77bf37f27a84b05f3234fef57650ad8d8c5ebfd445bb2bd9f7cdb722c/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/0c94fea46ae78d87acd6fb21d038d0b0e936b0a61f24dc2b96958fd0dbe02d68/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/7ad1b72a0e7085709549989525b7311692c1f83ebc8a97b9d24c54a776725fc9/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/bb4d457897cfff13b005b037c2f45e64c9334f548eb5892decf263d75be80499/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/2f99e132999b9bc1669dc254abbe6761ddea29e7ebb6fcf3777ef112659ad528/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/cd01f8597675182952b2b02d2190dd1367a0a0db5d4671bb7dc39f3521023962/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/1010210cb06944e325cbbe627c399e88bc730462435c29b850b4ae537d97618d/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/2249991ad7a41260e6f37945d47f3aeb8dd932c14cf864fff43c16e1f491ff01/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/e78857f4596f503ce5bca70172d1d746de29d6bf1c9ac82a158b9556a188a477/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/b48602c4bfdf62d7d9d1a87f493c53dc6d1761268c59ee4e343831ac921a8e79/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/76aad3021db38df239b607f7264f906e40973910080faf75b4f8eb9673d6d469/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/bca39b6595347aeeb5758a86f82ed1cfd2d1d9d13c56cf184426edda84fcb339/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/b961742c635b605fde77c95f2b683a758f81aa4c20ee1aa90d5f2cbafd0d7f47/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/bffb264146b5e480e5ad7689b06d2dc86183182fb8ea9bb119935c3f86687bcc/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/a6bf7867846365b9a54566af76aff5bc0fb1d97f7e7564a7a54ac7749e34b2f7/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/d22f3be5650820d883e2f5c52b65a3e77ca473fadfd58d280a81236e69f49251/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e1faabc6939bc11c373b5211ee15be673ec79dd9dd81b85fa295e0602f077d78/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b39399ab939cbeab0fecf3a745946ea2349ac00f6dda27f91631bb78268fb7e3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cae3e4d43467b03608532c7e4ca7812e4107f35863658757ecc0cf627fd85b88/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a3ab792b4ac08b4d6a5c9c2b6849b44376517c34021b91407eead4dc2b51257b/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/2c9ef41dcceac4d62f8ac308310a50e2b229e8cc0eca710f8800f54a5b4cf732/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/e1a782fc923c60e6d3fcb6200543816bfc58e3044269cbba69f0fc9b397b1514/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/354b1e849e81bcdf7a2c211431eecaa243d331a2197e528f43a32a1f6ca15140/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/93b7e41f096180ebc74fafeefedad75f4e6271383f2771236fa08248c3c285c5/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3ee721cedb5b1f446d856331d6eb9bc770b067a152c1c1bbe205370de5f41bc2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8b88a6397f1c72f8f2fd5bc86dee1f177c90bb91380ef2f9c8305809dade5bb1/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 08:32:56.988035 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.987347 5710 manager.go:217] Machine: {Timestamp:2026-11-01 08:32:56.986695387 +0000 UTC m=+0.075032049 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094035 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec259e04ad369ead7003a93869eeef76 SystemUUID:ec259e04-ad36-9ead-7003-a93869eeef76 BootID:56d7ba65-b81a-402b-9449-5371b404cc68 Filesystems:[{Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb599c9a-0381-45c2-801d-ef6578d0ea9d/volumes/kubernetes.io~projected/kube-api-access-qzggv DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e86830335230f0ba69e7b91e3da427aa64b91db3342477348d535416e13dbb31/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~projected/kube-api-access-zx9bq DeviceMajor:0 DeviceMinor:346 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e698209-ef38-47d5-be80-4e5208813720/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:394 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~projected/kube-api-access-d6ljf DeviceMajor:0 DeviceMinor:307 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9301cdec004affd1bbe02f7c8ad4204487bffd2a0b3a094c8793a083aaca3bf4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4bbf8093390fac9beef96506c0992eb5a50d61438abeafa36a56e9bfd21c48f/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~projected/kube-api-access-cmhh4 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ff83b640ff84de6e6daff0ea4e543e02a3f60ea0868b8da34b37efb176f1e49/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b175e2b21bdb2918c1ad5d18921e6ce41b0589809202a263aca3347bc9895f3b/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29716515-5846-432e-b2a2-e63ce1ea7059/volumes/kubernetes.io~projected/kube-api-access-ppxql DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/760c0d142c8198f4a10d673e550df4ac46dc36bec77e6e7006166e5853f0b846/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:301 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6611ffa7-eb5a-4a4f-be97-2d444e688e8d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff66af40d6ac5ae7a5455f18625503d4dfbafa553b3d4a29c84e279a4b227cb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e698209-ef38-47d5-be80-4e5208813720/volumes/kubernetes.io~projected/kube-api-access-526sp DeviceMajor:0 DeviceMinor:398 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volumes/kubernetes.io~projected/kube-api-access-mln9l DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f917c59e3c5a7fb00a6e53a3543cb9d3bf6c4252cbae99c426661c1ce713691c/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af1dcebb-bbfe-4e4a-b684-84012ddc57b8/volumes/kubernetes.io~projected/kube-api-access-rd87k DeviceMajor:0 DeviceMinor:370 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7d57cfa0-eef3-450d-9030-3049ee009711/volumes/kubernetes.io~projected/kube-api-access-76m8h DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15ee2bde-64f4-4580-8969-c395abf4c2f0/volumes/kubernetes.io~projected/kube-api-access-kb8nb DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:297 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~projected/kube-api-access-fcssx DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96d3c09554f8bbc57cc70a67593ebe91a64a34bbaa89c4bb2210a20eed65e473/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16f1c6a3-98c1-4970-9476-3330567de96c/volumes/kubernetes.io~projected/kube-api-access-fld4z DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47c5177d11b3fe8b6bf60267279f93e4a4fc4be683760482099b448c8e394f75/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/958489ac-f598-4365-a166-22371ca307d4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/388ff44e-1500-4858-87d0-7a33eee493d3/volumes/kubernetes.io~projected/kube-api-access-5k7rj DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~projected/kube-api-access-8gbk5 DeviceMajor:0 DeviceMinor:306 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0c527af-be39-4887-990a-0dd793e2a751/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8149186f748f6eda23aadc1a20784cb0e1c00721248e6d6cd7bcb3a00c4a3a1c/userdata/shm DeviceMajor:0 DeviceMinor:399 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/15ee2bde-64f4-4580-8969-c395abf4c2f0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d57cfa0-eef3-450d-9030-3049ee009711/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:362 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/6611ffa7-eb5a-4a4f-be97-2d444e688e8d/volumes/kubernetes.io~projected/kube-api-access-4b8mx DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/451acf74665a5dd41c12c2c5bf6f6b6eb212ddabcfb32c1d10829d803d4b798d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5470e32cc950b98577df7a9b8ef99148ba8a3298d299b4306062812bf035c54f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:415 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~projected/kube-api-access-mnx4n DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ad0ce105e8dbb89145a69e7c0f96bdc2fb11811460f4f81e9e28e5767e36f83/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e5459fb3d503ce9cd8bbff4c70fd66ab7513a751c4c916a8b52a9ec1a58b954/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e97a3a69-d419-495c-877f-651eae6172f7/volumes/kubernetes.io~projected/kube-api-access-57grl DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96b357ecac71ded6fe2e03cd8c72e36fcb231e01295793c15650420f1cb52c89/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29716515-5846-432e-b2a2-e63ce1ea7059/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7a28283c-3f57-4276-a7d1-a7792377f37b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:300 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/59c8058b-626b-4462-9765-5157b802d928/volumes/kubernetes.io~projected/kube-api-access-6j4fh DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22098de3-a796-47bb-98c9-30991caa95ac/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:265 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02b8bd3068ab4c469ba80f600108f92c23e6603b58bad91845c90d099bfda55a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes/kubernetes.io~projected/kube-api-access-7vf7g DeviceMajor:0 DeviceMinor:294 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/005b7e1777dabcb29ce1d03f46ee160a6d38e46d0e5007fde4acd36935acc1c8/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01a2f75dbdda4e391692761defea648ad6f201bc62977185c6cd09aaaba79c0f/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7df0abb7156d8cbc62503fd2cc8158056aafa2831037d91092d5513a6caa1ffc/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fc68d9cc-01a2-49b3-b0f6-f510d04caefa/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58067792-becb-4036-a80e-be1522d0f974/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36920425522d28bc9398a98f3b0c6be76a55746c9d5408ede89bdb70172fdd02/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:411 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ffc5727-de93-4b25-bd1c-46922c36a7e5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee31b8bb-4ebe-41dd-a701-66eaf42137da/volumes/kubernetes.io~projected/kube-api-access-cx75b DeviceMajor:0 DeviceMinor:299 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes/kubernetes.io~projected/kube-api-access-66p5s DeviceMajor:0 DeviceMinor:416 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a07fa048-d1fa-4b73-8120-6d04c02c91e9/volumes/kubernetes.io~projected/kube-api-access-cn7gx DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2aa7b442cae37cd MacAddress:16:65:14:a7:98:4f Speed:10000 Mtu:8901} {Name:2ff83b640ff84de MacAddress:96:43:f3:6f:9c:fd Speed:10000 Mtu:8901} {Name:3380373f718a0d4 MacAddress:ca:8b:8b:ac:ac:8d Speed:10000 Mtu:8901} {Name:36920425522d28b MacAddress:36:d8:61:b4:15:90 Speed:10000 Mtu:8901} {Name:47c5177d11b3fe8 MacAddress:4e:ed:f5:19:41:48 Speed:10000 Mtu:8901} {Name:652cd194b535826 MacAddress:9a:b9:a9:aa:db:3d Speed:10000 Mtu:8901} {Name:760c0d142c8198f MacAddress:7a:48:19:3c:1f:6c Speed:10000 Mtu:8901} {Name:7df0abb7156d8cb MacAddress:ba:ef:5a:11:50:03 Speed:10000 Mtu:8901} {Name:8149186f748f6ed MacAddress:fa:f4:20:1a:36:65 Speed:10000 Mtu:8901} {Name:96d3c09554f8bbc MacAddress:2a:f0:0f:d3:bf:a2 Speed:10000 Mtu:8901} {Name:9e5459fb3d503ce MacAddress:62:fc:59:2e:eb:fb Speed:10000 Mtu:8901} {Name:b175e2b21bdb291 MacAddress:ce:3b:35:16:60:f1 Speed:10000 Mtu:8901} {Name:b4bbf8093390fac MacAddress:b6:df:80:3c:87:78 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ca:c6:08:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:26:35:7e:72:0a:66 Speed:0 Mtu:8901} {Name:e86830335230f0b MacAddress:ba:44:1a:57:c6:6e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ca:c6:08:dd Speed:0 Mtu:9001} {Name:f917c59e3c5a7fb MacAddress:b6:d5:b2:5b:fd:ad Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:02:9d:41:ef:b0:dc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:42:78:a1:51:ae Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 08:32:56.988462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.988445 5710 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 08:32:56.988617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.988600 5710 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 08:32:56.989031 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.988999 5710 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 08:32:56.989180 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989026 5710 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-16.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 08:32:56.989245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989183 5710 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 08:32:56.989245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989195 5710 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 08:32:56.989245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989210 5710 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 08:32:56.989245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989229 5710 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 08:32:56.989418 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989397 5710 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 08:32:56.989666 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989651 5710 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 08:32:56.989835 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989822 5710 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 08:32:56.989896 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989876 5710 kubelet.go:493] "Attempting to sync node with API server" Nov 01 08:32:56.989945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989897 5710 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 08:32:56.989945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989913 5710 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 08:32:56.989945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989927 5710 kubelet.go:405] "Adding apiserver pod source" Nov 01 08:32:56.989945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.989940 5710 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 08:32:56.990680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.990664 5710 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 08:32:56.990680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.990677 5710 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 08:32:56.991360 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.991338 5710 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 08:32:56.995414 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.995395 5710 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 08:32:56.995414 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.995413 5710 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 08:32:56.995572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.995556 5710 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 08:32:56.996002 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.995972 5710 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 08:32:56.996131 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996020 5710 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996149 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996167 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996176 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996184 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996193 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996203 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996214 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996224 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996238 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 08:32:56.996258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996251 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 08:32:56.996593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996270 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 08:32:56.996593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996288 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 08:32:56.996593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996408 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 08:32:56.996593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:56.996422 5710 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 08:32:57.000589 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.000559 5710 server.go:1263] "Started kubelet" Nov 01 08:32:57.000707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.000643 5710 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 08:32:57.001428 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.000858 5710 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 08:32:57.001428 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.000953 5710 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 08:32:57.001428 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.001359 5710 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 08:32:57.001284 ip-10-0-143-16 systemd[1]: Started Kubernetes Kubelet. Nov 01 08:32:57.002146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.002126 5710 server.go:317] "Adding debug handlers to kubelet server" Nov 01 08:32:57.008563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008519 5710 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 08:32:57.008794 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008583 5710 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 08:32:57.008931 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.008877 5710 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:32:57.008931 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008874 5710 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 08:26:46 +0000 UTC" deadline="2028-04-30 05:03:26.938331366 +0000 UTC" Nov 01 08:32:57.008931 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008913 5710 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13100h30m29.929422081s" Nov 01 08:32:57.008931 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008920 5710 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 08:32:57.009153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008934 5710 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 08:32:57.009153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.008938 5710 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 08:32:57.012021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.011966 5710 factory.go:55] Registering systemd factory Nov 01 08:32:57.012021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.012018 5710 factory.go:223] Registration of the systemd container factory successfully Nov 01 08:32:57.014966 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.014914 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e698209-ef38-47d5-be80-4e5208813720" volumeName="kubernetes.io/secret/0e698209-ef38-47d5-be80-4e5208813720-monitoring-plugin-cert" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.014966 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.014983 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.014997 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d398bcb3-f7fc-4307-bbc3-6acccfcdc43c" volumeName="kubernetes.io/empty-dir/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-volume-directive-shadow" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015012 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015024 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015035 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015048 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffc5727-de93-4b25-bd1c-46922c36a7e5" volumeName="kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-tuned" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015065 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22098de3-a796-47bb-98c9-30991caa95ac" volumeName="kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-tls" seLinuxMountContext="" Nov 01 08:32:57.015080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015079 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22098de3-a796-47bb-98c9-30991caa95ac" volumeName="kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015091 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015105 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a28283c-3f57-4276-a7d1-a7792377f37b" volumeName="kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015116 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a28283c-3f57-4276-a7d1-a7792377f37b" volumeName="kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-tls" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015129 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015065 5710 factory.go:153] Registering CRI-O factory Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015156 5710 factory.go:223] Registration of the crio container factory successfully Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015143 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d398bcb3-f7fc-4307-bbc3-6acccfcdc43c" volumeName="kubernetes.io/projected/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-api-access-d6ljf" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015219 5710 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015226 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29716515-5846-432e-b2a2-e63ce1ea7059" volumeName="kubernetes.io/projected/29716515-5846-432e-b2a2-e63ce1ea7059-kube-api-access-ppxql" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015244 5710 factory.go:103] Registering Raw factory Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015249 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a28283c-3f57-4276-a7d1-a7792377f37b" volumeName="kubernetes.io/empty-dir/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-textfile" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015258 5710 manager.go:1199] Started watching for new ooms in manager Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015261 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015274 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb599c9a-0381-45c2-801d-ef6578d0ea9d" volumeName="kubernetes.io/projected/bb599c9a-0381-45c2-801d-ef6578d0ea9d-kube-api-access-qzggv" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015286 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015297 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015308 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31b8bb-4ebe-41dd-a701-66eaf42137da" volumeName="kubernetes.io/projected/ee31b8bb-4ebe-41dd-a701-66eaf42137da-kube-api-access-cx75b" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015321 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31b8bb-4ebe-41dd-a701-66eaf42137da" volumeName="kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015333 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015345 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015356 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22098de3-a796-47bb-98c9-30991caa95ac" volumeName="kubernetes.io/configmap/22098de3-a796-47bb-98c9-30991caa95ac-metrics-client-ca" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015368 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22098de3-a796-47bb-98c9-30991caa95ac" volumeName="kubernetes.io/projected/22098de3-a796-47bb-98c9-30991caa95ac-kube-api-access-mnx4n" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015379 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="958489ac-f598-4365-a166-22371ca307d4" volumeName="kubernetes.io/secret/958489ac-f598-4365-a166-22371ca307d4-agent-certs" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015398 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015413 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015424 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc68d9cc-01a2-49b3-b0f6-f510d04caefa" volumeName="kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015443 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f1c6a3-98c1-4970-9476-3330567de96c" volumeName="kubernetes.io/configmap/16f1c6a3-98c1-4970-9476-3330567de96c-serviceca" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015460 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015476 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d398bcb3-f7fc-4307-bbc3-6acccfcdc43c" volumeName="kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015489 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015501 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f1c6a3-98c1-4970-9476-3330567de96c" volumeName="kubernetes.io/projected/16f1c6a3-98c1-4970-9476-3330567de96c-kube-api-access-fld4z" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015513 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015524 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07fa048-d1fa-4b73-8120-6d04c02c91e9" volumeName="kubernetes.io/projected/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kube-api-access-cn7gx" seLinuxMountContext="" Nov 01 08:32:57.015521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015535 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d398bcb3-f7fc-4307-bbc3-6acccfcdc43c" volumeName="kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-tls" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015547 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31b8bb-4ebe-41dd-a701-66eaf42137da" volumeName="kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-tls" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015561 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ee2bde-64f4-4580-8969-c395abf4c2f0" volumeName="kubernetes.io/empty-dir/15ee2bde-64f4-4580-8969-c395abf4c2f0-data-volume" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015572 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015581 5710 manager.go:319] Starting recovery of all containers Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015585 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29716515-5846-432e-b2a2-e63ce1ea7059" volumeName="kubernetes.io/secret/29716515-5846-432e-b2a2-e63ce1ea7059-cert" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015598 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58067792-becb-4036-a80e-be1522d0f974" volumeName="kubernetes.io/secret/58067792-becb-4036-a80e-be1522d0f974-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015609 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d57cfa0-eef3-450d-9030-3049ee009711" volumeName="kubernetes.io/projected/7d57cfa0-eef3-450d-9030-3049ee009711-kube-api-access-76m8h" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015620 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0c527af-be39-4887-990a-0dd793e2a751" volumeName="kubernetes.io/secret/d0c527af-be39-4887-990a-0dd793e2a751-tls-certificates" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015632 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d398bcb3-f7fc-4307-bbc3-6acccfcdc43c" volumeName="kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-metrics-client-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015643 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015655 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c8058b-626b-4462-9765-5157b802d928" volumeName="kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-binary-copy" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015666 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6611ffa7-eb5a-4a4f-be97-2d444e688e8d" volumeName="kubernetes.io/secret/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-metrics-tls" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015687 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a28283c-3f57-4276-a7d1-a7792377f37b" volumeName="kubernetes.io/projected/7a28283c-3f57-4276-a7d1-a7792377f37b-kube-api-access-8gbk5" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015699 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffc5727-de93-4b25-bd1c-46922c36a7e5" volumeName="kubernetes.io/projected/9ffc5727-de93-4b25-bd1c-46922c36a7e5-kube-api-access-cmhh4" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015712 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015740 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015751 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015761 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c8058b-626b-4462-9765-5157b802d928" volumeName="kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015781 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e97a3a69-d419-495c-877f-651eae6172f7" volumeName="kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-multus-daemon-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015792 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffc5727-de93-4b25-bd1c-46922c36a7e5" volumeName="kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-tmp" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015804 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015815 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6611ffa7-eb5a-4a4f-be97-2d444e688e8d" volumeName="kubernetes.io/empty-dir/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-tmp-dir" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015826 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015837 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e97a3a69-d419-495c-877f-651eae6172f7" volumeName="kubernetes.io/projected/e97a3a69-d419-495c-877f-651eae6172f7-kube-api-access-57grl" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015848 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015858 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015870 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd38f917-f55b-458f-bfd4-18ea32a077af" volumeName="kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015883 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015897 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ee2bde-64f4-4580-8969-c395abf4c2f0" volumeName="kubernetes.io/configmap/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015911 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a28283c-3f57-4276-a7d1-a7792377f37b" volumeName="kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015922 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d57cfa0-eef3-450d-9030-3049ee009711" volumeName="kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015935 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="958489ac-f598-4365-a166-22371ca307d4" volumeName="kubernetes.io/configmap/958489ac-f598-4365-a166-22371ca307d4-konnectivity-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015949 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015963 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="388ff44e-1500-4858-87d0-7a33eee493d3" volumeName="kubernetes.io/projected/388ff44e-1500-4858-87d0-7a33eee493d3-kube-api-access-5k7rj" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015976 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee31b8bb-4ebe-41dd-a701-66eaf42137da" volumeName="kubernetes.io/configmap/ee31b8bb-4ebe-41dd-a701-66eaf42137da-metrics-client-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.015989 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016002 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bada36-7447-4f33-9279-ef2b7c1a0c33" volumeName="kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016015 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ee2bde-64f4-4580-8969-c395abf4c2f0" volumeName="kubernetes.io/secret/15ee2bde-64f4-4580-8969-c395abf4c2f0-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016029 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016042 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e698209-ef38-47d5-be80-4e5208813720" volumeName="kubernetes.io/projected/0e698209-ef38-47d5-be80-4e5208813720-kube-api-access-526sp" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016054 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15ee2bde-64f4-4580-8969-c395abf4c2f0" volumeName="kubernetes.io/projected/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-api-access-kb8nb" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016070 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58067792-becb-4036-a80e-be1522d0f974" volumeName="kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-env-overrides" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016084 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58067792-becb-4036-a80e-be1522d0f974" volumeName="kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016097 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016111 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="388ff44e-1500-4858-87d0-7a33eee493d3" volumeName="kubernetes.io/empty-dir/388ff44e-1500-4858-87d0-7a33eee493d3-tmp-dir" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016124 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58067792-becb-4036-a80e-be1522d0f974" volumeName="kubernetes.io/projected/58067792-becb-4036-a80e-be1522d0f974-kube-api-access-mln9l" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016136 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c8058b-626b-4462-9765-5157b802d928" volumeName="kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016149 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58067792-becb-4036-a80e-be1522d0f974" volumeName="kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-script-lib" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016162 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af1dcebb-bbfe-4e4a-b684-84012ddc57b8" volumeName="kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016175 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016195 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af4e88fb-f709-4c95-828d-9ef46ff8e159" volumeName="kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016208 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c8058b-626b-4462-9765-5157b802d928" volumeName="kubernetes.io/projected/59c8058b-626b-4462-9765-5157b802d928-kube-api-access-6j4fh" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016220 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6611ffa7-eb5a-4a4f-be97-2d444e688e8d" volumeName="kubernetes.io/configmap/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-config-volume" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016233 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6611ffa7-eb5a-4a4f-be97-2d444e688e8d" volumeName="kubernetes.io/projected/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-kube-api-access-4b8mx" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016245 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a28283c-3f57-4276-a7d1-a7792377f37b" volumeName="kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-metrics-client-ca" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016258 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb599c9a-0381-45c2-801d-ef6578d0ea9d" volumeName="kubernetes.io/configmap/bb599c9a-0381-45c2-801d-ef6578d0ea9d-iptables-alerter-script" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016272 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d398bcb3-f7fc-4307-bbc3-6acccfcdc43c" volumeName="kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016286 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e97a3a69-d419-495c-877f-651eae6172f7" volumeName="kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-cni-binary-copy" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016301 5710 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0580a40d-15af-4a92-86d4-eb3fa865b8a8" volumeName="kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config" seLinuxMountContext="" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016313 5710 reconstruct.go:97] "Volume reconstruction finished" Nov 01 08:32:57.017060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.016322 5710 reconciler.go:29] "Reconciler: start to sync state" Nov 01 08:32:57.047543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.047492 5710 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 08:32:57.050195 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.050168 5710 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 08:32:57.050195 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.050199 5710 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 08:32:57.050419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.050223 5710 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 08:32:57.050419 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.050264 5710 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 08:32:57.054097 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.052861 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:57.055548 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.055519 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:57.064988 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.064953 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" exitCode=0 Nov 01 08:32:57.069614 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.069585 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:32:57.070097 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.070069 5710 generic.go:356] "Generic (PLEG): container finished" podID="58067792-becb-4036-a80e-be1522d0f974" containerID="42c8d9328a047262c4f49963d221f8e96d9151b8d828319424e44f0d0e30c7f3" exitCode=1 Nov 01 08:32:57.083217 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.083123 5710 generic.go:356] "Generic (PLEG): container finished" podID="7a28283c-3f57-4276-a7d1-a7792377f37b" containerID="d5419398af8a5c58dfc0c77f86bbc8f9fa9be4b2db018a9fb4adad4ffac78a8a" exitCode=0 Nov 01 08:32:57.100124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.100086 5710 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="e7e3520c0f537b932e74b29471c723cdcce6509f0d3b0261b25cd7ec0f260181" exitCode=0 Nov 01 08:32:57.100124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.100117 5710 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="332c3cd0cfe6ade73967c91aad905ee4f6517b42b566874aa7eaf92ddf14f666" exitCode=0 Nov 01 08:32:57.100124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.100124 5710 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="c35541d399cf4784c584f3059c263a1401ba14194baabe6b52e83c03f80ef059" exitCode=0 Nov 01 08:32:57.100124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.100129 5710 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="82e4b379e2a3b4bcf8aa728afdd469bf9c654d4e3dc9c2122bb43b92a5281a1a" exitCode=0 Nov 01 08:32:57.100124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.100134 5710 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="dd3f35704f3cdfd9bce341cd3b651ceb82e845a2cd2e0a1ac8251ba0d8955a23" exitCode=0 Nov 01 08:32:57.100124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.100140 5710 generic.go:356] "Generic (PLEG): container finished" podID="59c8058b-626b-4462-9765-5157b802d928" containerID="83d637d063dac1935ac195375fc78294ef396de819530b4cc610ea4d70b37e63" exitCode=0 Nov 01 08:32:57.108953 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.108924 5710 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:32:57.118067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.118023 5710 generic.go:356] "Generic (PLEG): container finished" podID="a01f6a26e421fb5f2615ed4d9d59831f" containerID="39ac6264f44c61b1cf8899609bec5c4bd0dadcefb4422e81845348359ca190cd" exitCode=0 Nov 01 08:32:57.144537 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.144504 5710 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:32:57.146604 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.146572 5710 manager.go:324] Recovery completed Nov 01 08:32:57.150645 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.150597 5710 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 08:32:57.171886 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.171858 5710 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:32:57.174221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.174198 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:32:57.174331 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.174239 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:32:57.174331 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.174254 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:32:57.177019 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177000 5710 cpu_manager.go:226] "Starting" policy="none" Nov 01 08:32:57.177019 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177015 5710 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 08:32:57.177188 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177022 5710 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:32:57.177188 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177035 5710 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 08:32:57.177188 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177171 5710 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 08:32:57.177297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177181 5710 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 08:32:57.177297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177199 5710 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 08:32:57.177297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177204 5710 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 08:32:57.177297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177210 5710 policy_none.go:50] "Start" Nov 01 08:32:57.177297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177218 5710 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 08:32:57.177297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177227 5710 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 08:32:57.177502 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177390 5710 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 08:32:57.177502 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177398 5710 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 08:32:57.177502 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177404 5710 policy_none.go:44] "Start" Nov 01 08:32:57.177991 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.177972 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:32:57.178093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.178007 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:32:57.178093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.178021 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:32:57.182263 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.182227 5710 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 08:32:57.185202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185187 5710 manager.go:341] "Starting Device Plugin manager" Nov 01 08:32:57.185310 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.185222 5710 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 08:32:57.185310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185235 5710 server.go:87] "Starting device plugin registration server" Nov 01 08:32:57.185568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185553 5710 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 08:32:57.185633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185568 5710 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 08:32:57.185678 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185652 5710 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 08:32:57.185812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185775 5710 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 08:32:57.185884 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185822 5710 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 08:32:57.185884 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.185829 5710 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 08:32:57.186812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.186747 5710 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T08:32:57.185809674Z","UUID":"a168d202-3d7a-4f2c-a94b-e21a8e12eb6f","Handler":null,"Name":"","Endpoint":""} Nov 01 08:32:57.187786 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.187765 5710 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 08:32:57.187883 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.187812 5710 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-16.us-west-2.compute.internal\" not found" Nov 01 08:32:57.188283 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.188264 5710 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 08:32:57.188356 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.188304 5710 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 08:32:57.191078 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.191054 5710 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:32:57.191512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.191489 5710 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:32:57.191604 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.191544 5710 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 08:32:57.283923 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.283886 5710 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 08:32:57.285842 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.285814 5710 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 08:32:57.286999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.286979 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 08:32:57.287089 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.287013 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 08:32:57.287089 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.287024 5710 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 08:32:57.287089 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.287077 5710 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.351130 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.351074 5710 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal"] Nov 01 08:32:57.351897 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.351875 5710 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.352064 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.351922 5710 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.352064 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.351979 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerStarted","Data":"b898206dc5481bd1d3ed5c77c96d23fe6c08650f7faf489b0b552ae8e254ffcf"} Nov 01 08:32:57.352064 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.352034 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerDied","Data":"39ac6264f44c61b1cf8899609bec5c4bd0dadcefb4422e81845348359ca190cd"} Nov 01 08:32:57.352064 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.352048 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"a01f6a26e421fb5f2615ed4d9d59831f","Type":"ContainerStarted","Data":"ecf92e0660ebffcf238d818100e1a857b1a1f3d7209254ea564bda31d3172f74"} Nov 01 08:32:57.352202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.352074 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"61735862d7ea6a76be3182acee32ca86","Type":"ContainerStarted","Data":"f617abb16c2801874b0b61df6c5ee619e073fc5cce6e60b1e658dbcf7f009571"} Nov 01 08:32:57.352202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.352087 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" event={"ID":"61735862d7ea6a76be3182acee32ca86","Type":"ContainerStarted","Data":"92ab10d45689c3fbf177015104d9689a7e7e7454e80b1fd0202dfcf577fbb60b"} Nov 01 08:32:57.391309 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.391273 5710 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.391475 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.391364 5710 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.411153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.411119 5710 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 08:32:57.411313 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.411184 5710 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.411621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.411610 5710 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 08:32:57.411652 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:57.411642 5710 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.493949 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.493916 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.493949 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.493956 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.494212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.493984 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61735862d7ea6a76be3182acee32ca86-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"61735862d7ea6a76be3182acee32ca86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.595041 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.595009 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.595041 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.595044 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.595226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.595060 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61735862d7ea6a76be3182acee32ca86-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"61735862d7ea6a76be3182acee32ca86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.595226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.595114 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.595226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.595120 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a01f6a26e421fb5f2615ed4d9d59831f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"a01f6a26e421fb5f2615ed4d9d59831f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.595226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.595168 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61735862d7ea6a76be3182acee32ca86-config\") pod \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" (UID: \"61735862d7ea6a76be3182acee32ca86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:57.990863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:57.990825 5710 apiserver.go:52] "Watching apiserver" Nov 01 08:32:58.088849 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.088821 5710 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 08:32:58.089588 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.089565 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6958777449-f4bt5","kube-system/global-pull-secret-syncer-tg7cb","kube-system/konnectivity-agent-t58md","kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-2bmk2","openshift-ingress-canary/ingress-canary-5t549","openshift-monitoring/node-exporter-p7p9g","openshift-dns/node-resolver-4szqv","openshift-dns/dns-default-9tkwt","openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5","openshift-monitoring/prometheus-operator-9cd89b74c-vssf9","openshift-multus/multus-lxh88","openshift-image-registry/node-ca-tg27q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2","openshift-multus/network-metrics-daemon-gxwrb","openshift-cluster-node-tuning-operator/tuned-fb5tm","openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal","openshift-console/console-66fbd6bf9f-hrn97","openshift-multus/multus-additional-cni-plugins-8nckw","openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc","openshift-ovn-kubernetes/ovnkube-node-jvnzm","openshift-insights/insights-runtime-extractor-bm9bj","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6","openshift-console/console-8b9c7fbff-kdtfn","openshift-network-diagnostics/network-check-target-z7b55"] Nov 01 08:32:58.111793 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.111761 5710 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 08:32:58.132423 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.132393 5710 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:58.132598 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.132524 5710 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:58.140550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.140524 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxggb\"" Nov 01 08:32:58.141207 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.141182 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqp75\"" Nov 01 08:32:58.141359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.141281 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 08:32:58.141517 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.141492 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 08:32:58.141592 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.141560 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xx5dr\"" Nov 01 08:32:58.141644 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.141587 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xrtv\"" Nov 01 08:32:58.142013 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.141990 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cdn26\"" Nov 01 08:32:58.142221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.142076 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 08:32:58.142221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.142165 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jcrrb\"" Nov 01 08:32:58.142221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.142198 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 08:32:58.142320 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.142226 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xk5lv\"" Nov 01 08:32:58.143264 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.143232 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d96l\"" Nov 01 08:32:58.143320 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.143296 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hnxt\"" Nov 01 08:32:58.144038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144019 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.144120 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144063 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 08:32:58.144249 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144223 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.144316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144292 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.144594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144573 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.144662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144580 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.144662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144582 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.144782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144705 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 08:32:58.144782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144761 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.144852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144782 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.144852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144788 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 08:32:58.144852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144793 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 08:32:58.144852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144834 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.145008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144867 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 08:32:58.145008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144875 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 08:32:58.145008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144898 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.145008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144912 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.145008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.144938 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 08:32:58.145491 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.145473 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.198624 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198587 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:58.198624 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198631 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-system-cni-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198657 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58067792-becb-4036-a80e-be1522d0f974-ovn-node-metrics-cert\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198682 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0c527af-be39-4887-990a-0dd793e2a751-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2j2r6\" (UID: \"d0c527af-be39-4887-990a-0dd793e2a751\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198766 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198846 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-conf\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198878 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-registration-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198904 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-host\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.198937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198929 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-multus\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.199581 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.198991 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-hostroot\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.199692 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199618 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.199692 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199659 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-var-lib-kubelet\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.199808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199759 5710 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 08:32:58.199854 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:58.199827 5710 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:58.199897 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199848 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-tmp\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.199953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199756 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-tmp\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.199996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199954 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.200038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.199991 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.200038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200022 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-textfile\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.200144 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200127 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-tmp-dir\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.200193 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200169 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.200235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200200 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/958489ac-f598-4365-a166-22371ca307d4-agent-certs\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:32:58.200235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200212 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-textfile\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.200317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200232 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb599c9a-0381-45c2-801d-ef6578d0ea9d-host-slash\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.200317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200265 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-multus-certs\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.200317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200279 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58067792-becb-4036-a80e-be1522d0f974-ovn-node-metrics-cert\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.200317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200301 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.200494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200335 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ljf\" (UniqueName: \"kubernetes.io/projected/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-api-access-d6ljf\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.200494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200351 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-tmp-dir\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.200494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200389 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-systemd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.200494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200426 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-etc-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.200494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200466 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.200494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200493 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-root\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.200782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200528 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-metrics-client-ca\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.200782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200562 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fld4z\" (UniqueName: \"kubernetes.io/projected/16f1c6a3-98c1-4970-9476-3330567de96c-kube-api-access-fld4z\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.200782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200596 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-socket-dir-parent\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.200782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200634 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.200782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200682 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e698209-ef38-47d5-be80-4e5208813720-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:58.201164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200912 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8nb\" (UniqueName: \"kubernetes.io/projected/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-api-access-kb8nb\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.201164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200960 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.201164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.200995 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-systemd\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.201164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201023 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-bin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.201164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201068 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526sp\" (UniqueName: \"kubernetes.io/projected/0e698209-ef38-47d5-be80-4e5208813720-kube-api-access-526sp\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:58.201792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201446 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15ee2bde-64f4-4580-8969-c395abf4c2f0-data-volume\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.201792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201450 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/958489ac-f598-4365-a166-22371ca307d4-agent-certs\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:32:58.201792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201493 5710 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 08:32:58.201792 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:32:58.201541 5710 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal" Nov 01 08:32:58.201792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201584 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-kubelet-config\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.201792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201623 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.202045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201838 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-multus-daemon-config\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.202045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201903 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/15ee2bde-64f4-4580-8969-c395abf4c2f0-data-volume\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.202045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201946 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.202045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.201994 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.202045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202028 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-var-lib-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.202245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202057 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-dbus\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.202245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202091 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-sys\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.202245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202140 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-original-pull-secret\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.202245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202149 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-tls\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.202398 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202262 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.202398 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202308 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-config-volume\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.202398 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202343 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76m8h\" (UniqueName: \"kubernetes.io/projected/7d57cfa0-eef3-450d-9030-3049ee009711-kube-api-access-76m8h\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:58.202398 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202378 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.202560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202414 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.202560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202446 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.202560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202478 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.202560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202511 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.202560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202539 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388ff44e-1500-4858-87d0-7a33eee493d3-hosts-file\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.202777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202582 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.202777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202625 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f1c6a3-98c1-4970-9476-3330567de96c-host\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.202777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202664 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-bin\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.202777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202708 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbk5\" (UniqueName: \"kubernetes.io/projected/7a28283c-3f57-4276-a7d1-a7792377f37b-kube-api-access-8gbk5\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.202777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202765 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.202981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202778 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.202981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202799 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln9l\" (UniqueName: \"kubernetes.io/projected/58067792-becb-4036-a80e-be1522d0f974-kube-api-access-mln9l\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.202981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202849 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.202981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202882 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.202981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202914 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.202981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202951 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-node-log\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.202983 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203010 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-metrics-tls\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203042 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203075 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203110 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203137 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-run\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203170 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-k8s-cni-cncf-io\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203205 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57grl\" (UniqueName: \"kubernetes.io/projected/e97a3a69-d419-495c-877f-651eae6172f7-kube-api-access-57grl\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.203225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203211 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.203597 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203238 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.203597 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203281 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-config\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.203597 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203308 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-tuned\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.203597 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203394 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-tuned\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.203781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.203672 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-device-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.204667 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.204643 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.204869 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.204836 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-sys-fs\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.204947 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.204892 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-config\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.204947 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.204904 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15ee2bde-64f4-4580-8969-c395abf4c2f0-crio-socket\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.205054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.204947 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.205054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.204982 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.205054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205016 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.205247 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205109 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.205247 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205209 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.205343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205277 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.205343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205288 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.205343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205297 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.205453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205344 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388ff44e-1500-4858-87d0-7a33eee493d3-tmp-dir\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.205453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205438 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7rj\" (UniqueName: \"kubernetes.io/projected/388ff44e-1500-4858-87d0-7a33eee493d3-kube-api-access-5k7rj\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.205540 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205478 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388ff44e-1500-4858-87d0-7a33eee493d3-tmp-dir\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.205893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205868 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.205981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.205954 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-systemd-units\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206039 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206026 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-env-overrides\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206089 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206065 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.206140 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206125 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.206231 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206214 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-env-overrides\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206220 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-lib-modules\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.206273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206265 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-socket-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.206328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206300 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206357 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206336 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysconfig\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.206440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206361 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-kubernetes\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.206440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206394 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.206440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206431 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f1c6a3-98c1-4970-9476-3330567de96c-serviceca\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.206547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206481 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-netns\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206513 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-ovn\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206618 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206544 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.206672 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206641 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22098de3-a796-47bb-98c9-30991caa95ac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.206740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206692 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.206790 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206739 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7gx\" (UniqueName: \"kubernetes.io/projected/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kube-api-access-cn7gx\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.206790 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206769 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-log-socket\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.206900 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206798 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29716515-5846-432e-b2a2-e63ce1ea7059-cert\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:58.206900 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206824 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-sys\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.206900 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206852 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:58.207034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206930 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzggv\" (UniqueName: \"kubernetes.io/projected/bb599c9a-0381-45c2-801d-ef6578d0ea9d-kube-api-access-qzggv\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.207034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206959 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhh4\" (UniqueName: \"kubernetes.io/projected/9ffc5727-de93-4b25-bd1c-46922c36a7e5-kube-api-access-cmhh4\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.207034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.206990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.207034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207018 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.207204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207047 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-cnibin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.207204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207076 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.207204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207105 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.207204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207155 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d57cfa0-eef3-450d-9030-3049ee009711-metrics-certs\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:58.207204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207159 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.207407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207230 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-cni-binary-copy\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.207407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207287 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f1c6a3-98c1-4970-9476-3330567de96c-serviceca\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.207407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207288 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-kubelet\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.207407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207338 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-netd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.207407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207375 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-script-lib\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207449 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4fh\" (UniqueName: \"kubernetes.io/projected/59c8058b-626b-4462-9765-5157b802d928-kube-api-access-6j4fh\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207475 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/22098de3-a796-47bb-98c9-30991caa95ac-kube-api-access-mnx4n\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207493 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207514 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8mx\" (UniqueName: \"kubernetes.io/projected/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-kube-api-access-4b8mx\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207534 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207536 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/58067792-becb-4036-a80e-be1522d0f974-ovnkube-script-lib\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207563 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx75b\" (UniqueName: \"kubernetes.io/projected/ee31b8bb-4ebe-41dd-a701-66eaf42137da-kube-api-access-cx75b\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.207594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207591 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207619 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207645 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207672 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-conf-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207700 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15ee2bde-64f4-4580-8969-c395abf4c2f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207751 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-slash\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207765 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207781 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/958489ac-f598-4365-a166-22371ca307d4-konnectivity-ca\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207807 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-netns\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207845 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-etc-kubernetes\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207879 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207925 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-kubelet\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207956 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/958489ac-f598-4365-a166-22371ca307d4-konnectivity-ca\") pod \"konnectivity-agent-t58md\" (UID: \"958489ac-f598-4365-a166-22371ca307d4\") " pod="kube-system/konnectivity-agent-t58md" Nov 01 08:32:58.207967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207964 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.207984 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208012 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb599c9a-0381-45c2-801d-ef6578d0ea9d-iptables-alerter-script\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208043 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxql\" (UniqueName: \"kubernetes.io/projected/29716515-5846-432e-b2a2-e63ce1ea7059-kube-api-access-ppxql\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208157 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee31b8bb-4ebe-41dd-a701-66eaf42137da-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208236 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208235 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb599c9a-0381-45c2-801d-ef6578d0ea9d-iptables-alerter-script\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208260 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-wtmp\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208309 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208344 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-cnibin\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208389 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208421 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.208495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208481 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-system-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.208961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208536 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.208961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208562 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.208961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208613 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-modprobe-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.208961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.208647 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.226054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226021 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 08:32:58.226231 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226179 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 08:32:58.226231 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226208 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 08:32:58.226305 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226281 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 08:32:58.226366 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226334 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 08:32:58.226428 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226363 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 08:32:58.226473 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226428 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lnbwc\"" Nov 01 08:32:58.226705 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226687 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 08:32:58.226815 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226762 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 08:32:58.226870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226827 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 08:32:58.226870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226835 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sctgh\"" Nov 01 08:32:58.226870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226763 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 08:32:58.226870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226808 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h6qxd\"" Nov 01 08:32:58.227056 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.226936 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 08:32:58.227056 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227000 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 08:32:58.227147 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227054 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 08:32:58.227147 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227086 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 08:32:58.227147 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227114 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 08:32:58.227307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227155 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 08:32:58.227307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227182 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6hfzp\"" Nov 01 08:32:58.227307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227187 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24cps\"" Nov 01 08:32:58.227307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227279 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d5rrp\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227325 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227329 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227399 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48mlk\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227406 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mnx4s\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227410 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227428 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227445 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8k6cl\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227461 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qwt9q\"" Nov 01 08:32:58.227485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227464 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227492 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227499 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7q4h\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227499 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227506 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227538 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h4rfg\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227470 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29716515-5846-432e-b2a2-e63ce1ea7059-cert\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227470 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227599 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-l9vpz\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227659 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kmhw9\"" Nov 01 08:32:58.227817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227732 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 08:32:58.228073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227828 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.228073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227852 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.228073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.227975 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/15ee2bde-64f4-4580-8969-c395abf4c2f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.228878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.228860 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.228923 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.228874 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.228923 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.228879 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.228923 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.228911 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.230000 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.229981 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d0c527af-be39-4887-990a-0dd793e2a751-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2j2r6\" (UID: \"d0c527af-be39-4887-990a-0dd793e2a751\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:32:58.231116 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.231100 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.232230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.232216 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.232284 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.232270 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0e698209-ef38-47d5-be80-4e5208813720-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:58.232408 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.232395 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-tls\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.232462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.232450 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.233403 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.233390 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.233481 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.233460 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.233587 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.233574 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.234626 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.234608 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.235703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.235685 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-metrics-tls\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.235796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.235710 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.235796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.235710 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.236886 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.236867 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.236941 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.236925 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.238083 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.238066 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22098de3-a796-47bb-98c9-30991caa95ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.238132 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.238096 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee31b8bb-4ebe-41dd-a701-66eaf42137da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.244157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244084 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 08:32:58.244157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244110 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.244157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244122 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.244332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244159 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.244416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244399 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 08:32:58.244416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244409 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.244523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244401 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 08:32:58.244523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244452 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 08:32:58.244523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244473 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.244523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244473 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.244523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244483 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.244652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244560 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.244652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244569 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 08:32:58.244652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244647 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.244879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244791 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 08:32:58.244879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244830 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 08:32:58.244879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244845 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.244879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244854 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 08:32:58.244879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244866 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 08:32:58.245063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244844 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 08:32:58.245063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.244900 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 08:32:58.245271 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.245256 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 08:32:58.245306 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.245268 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 08:32:58.245508 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.245489 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.245592 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.245577 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.247615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.247597 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22098de3-a796-47bb-98c9-30991caa95ac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.247679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.247657 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-cni-binary-copy\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.247735 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.247657 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.247735 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.247658 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-binary-copy\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.247735 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.247707 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.248890 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.248876 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.249033 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.248995 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee31b8bb-4ebe-41dd-a701-66eaf42137da-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.249078 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.249035 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.251189 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.251173 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.251277 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.251186 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-metrics-client-ca\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.251329 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.251304 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.251826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.251805 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.252436 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.252413 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.252538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.252519 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e97a3a69-d419-495c-877f-651eae6172f7-multus-daemon-config\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.252587 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.252520 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59c8058b-626b-4462-9765-5157b802d928-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.252587 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.252553 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-config-volume\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.253729 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.253696 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.253769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.253697 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.253828 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.253814 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.254132 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.254116 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.273417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.273381 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.273606 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274095 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbk5\" (UniqueName: \"kubernetes.io/projected/7a28283c-3f57-4276-a7d1-a7792377f37b-kube-api-access-8gbk5\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274108 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76m8h\" (UniqueName: \"kubernetes.io/projected/7d57cfa0-eef3-450d-9030-3049ee009711-kube-api-access-76m8h\") pod \"network-metrics-daemon-gxwrb\" (UID: \"7d57cfa0-eef3-450d-9030-3049ee009711\") " pod="openshift-multus/network-metrics-daemon-gxwrb" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274096 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8nb\" (UniqueName: \"kubernetes.io/projected/15ee2bde-64f4-4580-8969-c395abf4c2f0-kube-api-access-kb8nb\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274303 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhh4\" (UniqueName: \"kubernetes.io/projected/9ffc5727-de93-4b25-bd1c-46922c36a7e5-kube-api-access-cmhh4\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274308 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln9l\" (UniqueName: \"kubernetes.io/projected/58067792-becb-4036-a80e-be1522d0f974-kube-api-access-mln9l\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274342 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ljf\" (UniqueName: \"kubernetes.io/projected/d398bcb3-f7fc-4307-bbc3-6acccfcdc43c-kube-api-access-d6ljf\") pod \"kube-state-metrics-5dc8d7d89b-vv4sc\" (UID: \"d398bcb3-f7fc-4307-bbc3-6acccfcdc43c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vv4sc" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274493 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxql\" (UniqueName: \"kubernetes.io/projected/29716515-5846-432e-b2a2-e63ce1ea7059-kube-api-access-ppxql\") pod \"ingress-canary-5t549\" (UID: \"29716515-5846-432e-b2a2-e63ce1ea7059\") " pod="openshift-ingress-canary/ingress-canary-5t549" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274832 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzggv\" (UniqueName: \"kubernetes.io/projected/bb599c9a-0381-45c2-801d-ef6578d0ea9d-kube-api-access-qzggv\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274844 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7rj\" (UniqueName: \"kubernetes.io/projected/388ff44e-1500-4858-87d0-7a33eee493d3-kube-api-access-5k7rj\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274881 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274881 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274925 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/22098de3-a796-47bb-98c9-30991caa95ac-kube-api-access-mnx4n\") pod \"prometheus-operator-9cd89b74c-vssf9\" (UID: \"22098de3-a796-47bb-98c9-30991caa95ac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vssf9" Nov 01 08:32:58.274970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.274926 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7gx\" (UniqueName: \"kubernetes.io/projected/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kube-api-access-cn7gx\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275065 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57grl\" (UniqueName: \"kubernetes.io/projected/e97a3a69-d419-495c-877f-651eae6172f7-kube-api-access-57grl\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275159 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4fh\" (UniqueName: \"kubernetes.io/projected/59c8058b-626b-4462-9765-5157b802d928-kube-api-access-6j4fh\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275360 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8mx\" (UniqueName: \"kubernetes.io/projected/6611ffa7-eb5a-4a4f-be97-2d444e688e8d-kube-api-access-4b8mx\") pod \"dns-default-9tkwt\" (UID: \"6611ffa7-eb5a-4a4f-be97-2d444e688e8d\") " pod="openshift-dns/dns-default-9tkwt" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275364 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fld4z\" (UniqueName: \"kubernetes.io/projected/16f1c6a3-98c1-4970-9476-3330567de96c-kube-api-access-fld4z\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275521 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") pod \"console-6958777449-f4bt5\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275529 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275559 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") pod \"console-8b9c7fbff-kdtfn\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275791 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526sp\" (UniqueName: \"kubernetes.io/projected/0e698209-ef38-47d5-be80-4e5208813720-kube-api-access-526sp\") pod \"monitoring-plugin-77f4f76594-hcmzn\" (UID: \"0e698209-ef38-47d5-be80-4e5208813720\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:32:58.275899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.275899 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd87k\" (UniqueName: \"kubernetes.io/projected/af1dcebb-bbfe-4e4a-b684-84012ddc57b8-kube-api-access-rd87k\") pod \"network-check-target-z7b55\" (UID: \"af1dcebb-bbfe-4e4a-b684-84012ddc57b8\") " pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:32:58.276543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.276526 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx75b\" (UniqueName: \"kubernetes.io/projected/ee31b8bb-4ebe-41dd-a701-66eaf42137da-kube-api-access-cx75b\") pod \"openshift-state-metrics-7bd7489957-pd9k5\" (UID: \"ee31b8bb-4ebe-41dd-a701-66eaf42137da\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pd9k5" Nov 01 08:32:58.277269 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.277253 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:32:58.277706 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.277690 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") pod \"console-66fbd6bf9f-hrn97\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:32:58.309908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309869 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-cnibin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309916 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-kubelet\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309940 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-netd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309972 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-conf-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309994 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-slash\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309999 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-kubelet\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310038 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-conf-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310055 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-netns\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.309999 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-cnibin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310074 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-slash\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310056 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-netd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310016 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-netns\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310131 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-etc-kubernetes\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310168 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-kubelet\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310184 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310204 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-wtmp\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310224 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-etc-kubernetes\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310245 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-kubelet\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310226 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310270 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310276 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-node-exporter-wtmp\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310250 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310292 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-cnibin\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310320 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-system-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310335 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-cnibin\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310346 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-modprobe-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310383 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-system-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310385 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-system-cni-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310418 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59c8058b-626b-4462-9765-5157b802d928-system-cni-dir\") pod \"multus-additional-cni-plugins-8nckw\" (UID: \"59c8058b-626b-4462-9765-5157b802d928\") " pod="openshift-multus/multus-additional-cni-plugins-8nckw" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310422 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-conf\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310454 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-registration-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310464 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-modprobe-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310479 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-host\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310493 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-registration-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310506 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-multus\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310512 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-conf\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310527 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-host\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310530 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-hostroot\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310546 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-multus\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310551 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-hostroot\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310561 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310590 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-var-lib-kubelet\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.310601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310619 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb599c9a-0381-45c2-801d-ef6578d0ea9d-host-slash\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310641 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-multus-certs\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310645 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310666 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-var-lib-kubelet\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310668 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-systemd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310683 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb599c9a-0381-45c2-801d-ef6578d0ea9d-host-slash\") pod \"iptables-alerter-2bmk2\" (UID: \"bb599c9a-0381-45c2-801d-ef6578d0ea9d\") " pod="openshift-network-operator/iptables-alerter-2bmk2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310702 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-systemd\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310708 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-multus-certs\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310712 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-etc-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310758 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310777 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-etc-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310820 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310847 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-root\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310875 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-socket-dir-parent\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310883 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-root\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310898 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-systemd\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310917 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-bin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310925 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-socket-dir-parent\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310955 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-systemd\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310978 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-kubelet-config\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311001 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-var-lib-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311015 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-dbus\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311027 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-kubelet-config\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.310979 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-var-lib-cni-bin\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311033 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-sys\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311075 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc68d9cc-01a2-49b3-b0f6-f510d04caefa-dbus\") pod \"global-pull-secret-syncer-tg7cb\" (UID: \"fc68d9cc-01a2-49b3-b0f6-f510d04caefa\") " pod="kube-system/global-pull-secret-syncer-tg7cb" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311080 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-sys\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311091 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388ff44e-1500-4858-87d0-7a33eee493d3-hosts-file\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311099 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-var-lib-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311114 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f1c6a3-98c1-4970-9476-3330567de96c-host\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311131 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-bin\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311153 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-node-log\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311153 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388ff44e-1500-4858-87d0-7a33eee493d3-hosts-file\") pod \"node-resolver-4szqv\" (UID: \"388ff44e-1500-4858-87d0-7a33eee493d3\") " pod="openshift-dns/node-resolver-4szqv" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311183 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311185 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f1c6a3-98c1-4970-9476-3330567de96c-host\") pod \"node-ca-tg27q\" (UID: \"16f1c6a3-98c1-4970-9476-3330567de96c\") " pod="openshift-image-registry/node-ca-tg27q" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311197 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-node-log\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311206 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-run\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311209 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-cni-bin\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311228 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-k8s-cni-cncf-io\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311266 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-device-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311266 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-run\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311282 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-host-run-k8s-cni-cncf-io\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311308 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysctl-d\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311309 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-device-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311341 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-sys-fs\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311365 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15ee2bde-64f4-4580-8969-c395abf4c2f0-crio-socket\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311389 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-sys-fs\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311392 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311419 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-systemd-units\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311435 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/15ee2bde-64f4-4580-8969-c395abf4c2f0-crio-socket\") pod \"insights-runtime-extractor-bm9bj\" (UID: \"15ee2bde-64f4-4580-8969-c395abf4c2f0\") " pod="openshift-insights/insights-runtime-extractor-bm9bj" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311442 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-lib-modules\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311454 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e97a3a69-d419-495c-877f-651eae6172f7-multus-cni-dir\") pod \"multus-lxh88\" (UID: \"e97a3a69-d419-495c-877f-651eae6172f7\") " pod="openshift-multus/multus-lxh88" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311461 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-systemd-units\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311468 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-socket-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311494 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311515 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysconfig\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311524 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-lib-modules\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311536 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-kubernetes\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311548 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-openvswitch\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311556 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-socket-dir\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311562 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-netns\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311590 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-kubernetes\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311590 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-ovn\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311623 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-run-ovn\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311618 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311568 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ffc5727-de93-4b25-bd1c-46922c36a7e5-etc-sysconfig\") pod \"tuned-fb5tm\" (UID: \"9ffc5727-de93-4b25-bd1c-46922c36a7e5\") " pod="openshift-cluster-node-tuning-operator/tuned-fb5tm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311651 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-host-run-netns\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311666 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-log-socket\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311679 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a07fa048-d1fa-4b73-8120-6d04c02c91e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-q66m2\" (UID: \"a07fa048-d1fa-4b73-8120-6d04c02c91e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q66m2" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311730 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/58067792-becb-4036-a80e-be1522d0f974-log-socket\") pod \"ovnkube-node-jvnzm\" (UID: \"58067792-becb-4036-a80e-be1522d0f974\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311759 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-sys\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.311953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.311818 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a28283c-3f57-4276-a7d1-a7792377f37b-sys\") pod \"node-exporter-p7p9g\" (UID: \"7a28283c-3f57-4276-a7d1-a7792377f37b\") " pod="openshift-monitoring/node-exporter-p7p9g" Nov 01 08:32:58.429230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.429197 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b9c7fbff-kdtfn"] Nov 01 08:32:58.872646 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:32:58.872601 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9tkwt" Nov 01 08:33:00.065286 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:00.065183 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:33:00.069234 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:00.069209 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcmzn" Nov 01 08:33:02.594466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:02.594436 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:33:03.932589 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:03.932548 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:33:03.933083 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:03.932819 5710 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 08:33:04.733041 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:04.733010 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2j2r6" Nov 01 08:33:05.073713 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:05.073623 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:33:05.784509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:05.784450 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:33:06.292588 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:06.292552 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:33:08.141479 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:08.141446 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:33:15.079061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:15.079028 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:33:15.083062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:15.083037 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:33:15.477153 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:15.477117 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:33:21.243958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:21.243929 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t58md" Nov 01 08:33:22.298865 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:22.298833 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvnzm" Nov 01 08:33:23.120904 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.120860 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66fbd6bf9f-hrn97" podUID="af4e88fb-f709-4c95-828d-9ef46ff8e159" containerName="console" containerID="cri-o://d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6" gracePeriod=15 Nov 01 08:33:23.386186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.386159 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:33:23.525166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525120 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525213 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525265 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525285 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525328 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525359 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525386 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525433 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") pod \"af4e88fb-f709-4c95-828d-9ef46ff8e159\" (UID: \"af4e88fb-f709-4c95-828d-9ef46ff8e159\") " Nov 01 08:33:23.525697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525519 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:33:23.525697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525676 5710 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af4e88fb-f709-4c95-828d-9ef46ff8e159-tmp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.525885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525712 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:23.525885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525780 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:23.525989 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525822 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:23.526041 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.525985 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:23.528076 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.528043 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:23.528176 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.528120 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:23.528176 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.528154 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g" pod "af4e88fb-f709-4c95-828d-9ef46ff8e159" (UID: "af4e88fb-f709-4c95-828d-9ef46ff8e159"). InnerVolumeSpecName "kube-api-access-7vf7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626750 5710 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626796 5710 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-oauth-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626808 5710 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-trusted-ca-bundle\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626817 5710 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626827 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vf7g\" (UniqueName: \"kubernetes.io/projected/af4e88fb-f709-4c95-828d-9ef46ff8e159-kube-api-access-7vf7g\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626837 5710 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e88fb-f709-4c95-828d-9ef46ff8e159-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:23.626847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:23.626847 5710 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af4e88fb-f709-4c95-828d-9ef46ff8e159-console-oauth-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.159632 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.159580 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8b9c7fbff-kdtfn" podUID="0580a40d-15af-4a92-86d4-eb3fa865b8a8" containerName="console" containerID="cri-o://7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e" gracePeriod=15 Nov 01 08:33:24.229088 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.229053 5710 generic.go:356] "Generic (PLEG): container finished" podID="af4e88fb-f709-4c95-828d-9ef46ff8e159" containerID="d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6" exitCode=0 Nov 01 08:33:24.229257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.229104 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fbd6bf9f-hrn97" Nov 01 08:33:24.229257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.229125 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fbd6bf9f-hrn97" event={"ID":"af4e88fb-f709-4c95-828d-9ef46ff8e159","Type":"ContainerDied","Data":"d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6"} Nov 01 08:33:24.229257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.229166 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fbd6bf9f-hrn97" event={"ID":"af4e88fb-f709-4c95-828d-9ef46ff8e159","Type":"ContainerDied","Data":"3380373f718a0d4980b6c6297511103fd0c0212145b3912a7db5a12440f15871"} Nov 01 08:33:24.229257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.229184 5710 scope.go:122] "RemoveContainer" containerID="d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6" Nov 01 08:33:24.271454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.271433 5710 scope.go:122] "RemoveContainer" containerID="d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6" Nov 01 08:33:24.271909 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:24.271881 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6\": container with ID starting with d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6 not found: ID does not exist" containerID="d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6" Nov 01 08:33:24.271973 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.271924 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6"} err="failed to get container status \"d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6\": rpc error: code = NotFound desc = could not find container \"d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6\": container with ID starting with d94ace207feea5fe7c521837559836282fc4f286ac81edbeef955e20f00c17c6 not found: ID does not exist" Nov 01 08:33:24.382165 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.382129 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fbd6bf9f-hrn97"] Nov 01 08:33:24.406517 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.406493 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:33:24.428100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.428031 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66fbd6bf9f-hrn97"] Nov 01 08:33:24.535121 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535087 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535337 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535157 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535337 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535196 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535337 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535255 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535379 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535460 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535533 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") pod \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\" (UID: \"0580a40d-15af-4a92-86d4-eb3fa865b8a8\") " Nov 01 08:33:24.535679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535627 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:24.535679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535635 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:24.535800 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535681 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:24.535937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535876 5710 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.535937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535902 5710 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-oauth-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.535937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535916 5710 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0580a40d-15af-4a92-86d4-eb3fa865b8a8-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.535937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.535920 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:33:24.538142 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.538109 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:24.538267 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.538169 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "kube-api-access-fcssx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:33:24.538267 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.538200 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config" pod "0580a40d-15af-4a92-86d4-eb3fa865b8a8" (UID: "0580a40d-15af-4a92-86d4-eb3fa865b8a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:24.636467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.636422 5710 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.636467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.636458 5710 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0580a40d-15af-4a92-86d4-eb3fa865b8a8-tmp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.636467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.636468 5710 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0580a40d-15af-4a92-86d4-eb3fa865b8a8-console-oauth-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:24.636467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:24.636478 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcssx\" (UniqueName: \"kubernetes.io/projected/0580a40d-15af-4a92-86d4-eb3fa865b8a8-kube-api-access-fcssx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:25.055134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.055104 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4e88fb-f709-4c95-828d-9ef46ff8e159" path="/var/lib/kubelet/pods/af4e88fb-f709-4c95-828d-9ef46ff8e159/volumes" Nov 01 08:33:25.235175 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.235145 5710 generic.go:356] "Generic (PLEG): container finished" podID="0580a40d-15af-4a92-86d4-eb3fa865b8a8" containerID="7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e" exitCode=0 Nov 01 08:33:25.235365 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.235189 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b9c7fbff-kdtfn" event={"ID":"0580a40d-15af-4a92-86d4-eb3fa865b8a8","Type":"ContainerDied","Data":"7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e"} Nov 01 08:33:25.235365 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.235232 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b9c7fbff-kdtfn" Nov 01 08:33:25.235365 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.235253 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b9c7fbff-kdtfn" event={"ID":"0580a40d-15af-4a92-86d4-eb3fa865b8a8","Type":"ContainerDied","Data":"2aa7b442cae37cddb06c94f2711b9aeb21889517297e993bc6aa3f86d092a692"} Nov 01 08:33:25.235365 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.235275 5710 scope.go:122] "RemoveContainer" containerID="7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e" Nov 01 08:33:25.243295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.243273 5710 scope.go:122] "RemoveContainer" containerID="7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e" Nov 01 08:33:25.243605 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:25.243585 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e\": container with ID starting with 7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e not found: ID does not exist" containerID="7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e" Nov 01 08:33:25.243657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.243615 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e"} err="failed to get container status \"7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e\": rpc error: code = NotFound desc = could not find container \"7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e\": container with ID starting with 7e058a771ca1b0a172ae5a09235d7462b042b5da25c0ab24e862ee63db04f86e not found: ID does not exist" Nov 01 08:33:25.387766 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.387731 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b9c7fbff-kdtfn"] Nov 01 08:33:25.433974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:25.433942 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b9c7fbff-kdtfn"] Nov 01 08:33:26.296058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:26.296028 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7b55" Nov 01 08:33:27.055954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:27.055916 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0580a40d-15af-4a92-86d4-eb3fa865b8a8" path="/var/lib/kubelet/pods/0580a40d-15af-4a92-86d4-eb3fa865b8a8/volumes" Nov 01 08:33:41.563688 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.563650 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:33:41.564255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.564009 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="alertmanager" containerID="cri-o://cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" gracePeriod=120 Nov 01 08:33:41.564255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.564044 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-metric" containerID="cri-o://86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" gracePeriod=120 Nov 01 08:33:41.564255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.564088 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-web" containerID="cri-o://59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" gracePeriod=120 Nov 01 08:33:41.564255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.564111 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="config-reloader" containerID="cri-o://0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" gracePeriod=120 Nov 01 08:33:41.564255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.564139 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="prom-label-proxy" containerID="cri-o://ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" gracePeriod=120 Nov 01 08:33:41.564255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.564144 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy" containerID="cri-o://110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" gracePeriod=120 Nov 01 08:33:41.824167 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.824083 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:41.975508 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975457 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975508 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975523 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975545 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975614 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975673 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975703 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975778 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.975831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975807 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.976123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975846 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.976123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975885 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.976123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975920 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.976123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.975965 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.976123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.976051 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") pod \"dd38f917-f55b-458f-bfd4-18ea32a077af\" (UID: \"dd38f917-f55b-458f-bfd4-18ea32a077af\") " Nov 01 08:33:41.977011 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.976935 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:33:41.977406 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.977375 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:41.977488 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.977405 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:33:41.978956 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.978893 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:41.979068 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.978996 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:41.979129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.979078 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:41.979383 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.979341 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:33:41.979503 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.979459 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:41.979575 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.979531 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:41.979739 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.979700 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:33:41.980934 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.980909 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "kube-api-access-zx9bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:33:41.984772 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.984688 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:41.991224 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:41.991188 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config" pod "dd38f917-f55b-458f-bfd4-18ea32a077af" (UID: "dd38f917-f55b-458f-bfd4-18ea32a077af"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077545 5710 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-config-out\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077578 5710 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077588 5710 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-main-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077597 5710 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077607 5710 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077617 5710 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-main-db\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.077625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077626 5710 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.078006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077637 5710 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-web-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.078006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077647 5710 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dd38f917-f55b-458f-bfd4-18ea32a077af-cluster-tls-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.078006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077655 5710 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.078006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077664 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd38f917-f55b-458f-bfd4-18ea32a077af-metrics-client-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.078006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077674 5710 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-tls-assets\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.078006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.077682 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9bq\" (UniqueName: \"kubernetes.io/projected/dd38f917-f55b-458f-bfd4-18ea32a077af-kube-api-access-zx9bq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:33:42.300440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300403 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" exitCode=0 Nov 01 08:33:42.300440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300433 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" exitCode=0 Nov 01 08:33:42.300440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300442 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" exitCode=0 Nov 01 08:33:42.300440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300448 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" exitCode=0 Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300454 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" exitCode=0 Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300462 5710 generic.go:356] "Generic (PLEG): container finished" podID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" exitCode=0 Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300487 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300483 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300622 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300650 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300676 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300687 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.300730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300710 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} Nov 01 08:33:42.301063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300757 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} Nov 01 08:33:42.301063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.300790 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dd38f917-f55b-458f-bfd4-18ea32a077af","Type":"ContainerDied","Data":"652cd194b53582608a4f5f4053e6bfac63e5f84231144c430d87980618ee1bbf"} Nov 01 08:33:42.309586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.309567 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.317027 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.316995 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.324259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.324238 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.331868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.331848 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.339641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.339622 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.347146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.347120 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.354882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.354862 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.355184 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.355164 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.355261 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.355192 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} err="failed to get container status \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" Nov 01 08:33:42.355261 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.355210 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.355481 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.355463 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.355524 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.355488 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} err="failed to get container status \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" Nov 01 08:33:42.355524 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.355504 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.355775 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.355755 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.355836 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.355782 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} err="failed to get container status \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" Nov 01 08:33:42.355836 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.355803 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.356076 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.356060 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.356123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356081 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} err="failed to get container status \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" Nov 01 08:33:42.356123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356095 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.356328 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.356307 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.356375 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356332 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} err="failed to get container status \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" Nov 01 08:33:42.356375 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356344 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.356580 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.356564 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.356623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356585 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} err="failed to get container status \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" Nov 01 08:33:42.356623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356599 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.356936 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:42.356917 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.356974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356939 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} err="failed to get container status \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" Nov 01 08:33:42.356974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.356950 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.357177 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357160 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} err="failed to get container status \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" Nov 01 08:33:42.357218 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357177 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.357410 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357391 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} err="failed to get container status \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" Nov 01 08:33:42.357451 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357411 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.357641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357620 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} err="failed to get container status \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" Nov 01 08:33:42.357683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357642 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.357876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357859 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} err="failed to get container status \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" Nov 01 08:33:42.357876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.357875 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.358084 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358068 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} err="failed to get container status \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" Nov 01 08:33:42.358084 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358085 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.358281 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358266 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} err="failed to get container status \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" Nov 01 08:33:42.358322 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358283 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.358506 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358489 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} err="failed to get container status \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" Nov 01 08:33:42.358547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358507 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.358751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358709 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} err="failed to get container status \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" Nov 01 08:33:42.358751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358748 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.358974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358957 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} err="failed to get container status \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" Nov 01 08:33:42.359021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.358974 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.359210 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359191 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} err="failed to get container status \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" Nov 01 08:33:42.359210 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359210 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.359421 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359402 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} err="failed to get container status \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" Nov 01 08:33:42.359476 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359421 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.359642 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359625 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} err="failed to get container status \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" Nov 01 08:33:42.359690 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359642 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.359882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359861 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} err="failed to get container status \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" Nov 01 08:33:42.359937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.359883 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.360137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360119 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} err="failed to get container status \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" Nov 01 08:33:42.360212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360139 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.360400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360383 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} err="failed to get container status \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" Nov 01 08:33:42.360446 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360400 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.360591 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360573 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} err="failed to get container status \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" Nov 01 08:33:42.360659 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360591 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.360958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360940 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} err="failed to get container status \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" Nov 01 08:33:42.361014 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.360958 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.361171 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361153 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} err="failed to get container status \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" Nov 01 08:33:42.361211 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361171 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.361374 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361357 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} err="failed to get container status \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" Nov 01 08:33:42.361440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361375 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.361610 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361593 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} err="failed to get container status \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" Nov 01 08:33:42.361656 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361611 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.361858 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361839 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} err="failed to get container status \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" Nov 01 08:33:42.361924 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.361858 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.362087 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362070 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} err="failed to get container status \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" Nov 01 08:33:42.362136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362088 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.362290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362273 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} err="failed to get container status \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" Nov 01 08:33:42.362358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362291 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.362514 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362497 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} err="failed to get container status \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" Nov 01 08:33:42.362556 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362515 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.362735 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362698 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} err="failed to get container status \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" Nov 01 08:33:42.362735 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362728 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.362973 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362956 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} err="failed to get container status \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" Nov 01 08:33:42.363018 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.362974 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.363187 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363171 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} err="failed to get container status \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" Nov 01 08:33:42.363254 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363188 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.363434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363415 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} err="failed to get container status \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" Nov 01 08:33:42.363479 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363434 5710 scope.go:122] "RemoveContainer" containerID="ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5" Nov 01 08:33:42.363634 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363616 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5"} err="failed to get container status \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": rpc error: code = NotFound desc = could not find container \"ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5\": container with ID starting with ff0e12a530f97338aeb61a0d4fe0a114e370b13bf177dca88f78ecf33f7324e5 not found: ID does not exist" Nov 01 08:33:42.363700 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363635 5710 scope.go:122] "RemoveContainer" containerID="86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0" Nov 01 08:33:42.363859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363842 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0"} err="failed to get container status \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": rpc error: code = NotFound desc = could not find container \"86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0\": container with ID starting with 86004f9a3e5ec55ab967739c6e7dc6a590cf41030ebac27f4e4e0dd16d7223d0 not found: ID does not exist" Nov 01 08:33:42.363901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.363859 5710 scope.go:122] "RemoveContainer" containerID="110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2" Nov 01 08:33:42.364070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364052 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2"} err="failed to get container status \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": rpc error: code = NotFound desc = could not find container \"110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2\": container with ID starting with 110e94ae568ce33fc62cde74bf3a16a7cf56738ff90a7f5244de2d95fc6e8df2 not found: ID does not exist" Nov 01 08:33:42.364137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364070 5710 scope.go:122] "RemoveContainer" containerID="59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7" Nov 01 08:33:42.364288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364268 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7"} err="failed to get container status \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": rpc error: code = NotFound desc = could not find container \"59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7\": container with ID starting with 59e1b79291aa95fdea70cc14a1b2ab759b2ea2f201aaf2dbbd087498772ed1a7 not found: ID does not exist" Nov 01 08:33:42.364332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364289 5710 scope.go:122] "RemoveContainer" containerID="0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410" Nov 01 08:33:42.364480 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364463 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410"} err="failed to get container status \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": rpc error: code = NotFound desc = could not find container \"0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410\": container with ID starting with 0f73ed93b139d1be6f94d06e29978fe66656aeabbec250e6fa7cf69cfa297410 not found: ID does not exist" Nov 01 08:33:42.364543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364481 5710 scope.go:122] "RemoveContainer" containerID="cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a" Nov 01 08:33:42.364695 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364676 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a"} err="failed to get container status \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": rpc error: code = NotFound desc = could not find container \"cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a\": container with ID starting with cce5283ae4d6597f46add35943015a1905b428bd78d418cc6d47069e3737526a not found: ID does not exist" Nov 01 08:33:42.364763 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364696 5710 scope.go:122] "RemoveContainer" containerID="96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0" Nov 01 08:33:42.364908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.364891 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0"} err="failed to get container status \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": rpc error: code = NotFound desc = could not find container \"96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0\": container with ID starting with 96e3d41ff875aa2b75f9f6ab7b76d15a4261bb5a33eed4d1c058faf7863eb6b0 not found: ID does not exist" Nov 01 08:33:42.456705 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.456668 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:33:42.503278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503241 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:33:42.503582 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503560 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:33:42.503884 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503870 5710 cpu_manager.go:414] "containerMap: removing container" podUID="af4e88fb-f709-4c95-828d-9ef46ff8e159" containerName="console" Nov 01 08:33:42.503925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503885 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4e88fb-f709-4c95-828d-9ef46ff8e159" containerName="console" Nov 01 08:33:42.503925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503897 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="prom-label-proxy" Nov 01 08:33:42.503925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503921 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="prom-label-proxy" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503929 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="alertmanager" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503934 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="alertmanager" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503943 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503947 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503955 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="init-config-reloader" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503959 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="init-config-reloader" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503968 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-metric" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503972 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-metric" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503978 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="config-reloader" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503982 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="config-reloader" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503992 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-web" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.503996 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-web" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504008 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0580a40d-15af-4a92-86d4-eb3fa865b8a8" containerName="console" Nov 01 08:33:42.504012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504012 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0580a40d-15af-4a92-86d4-eb3fa865b8a8" containerName="console" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504062 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="init-config-reloader" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504071 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-metric" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504076 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="config-reloader" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504082 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy-web" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504088 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0580a40d-15af-4a92-86d4-eb3fa865b8a8" containerName="console" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504094 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4e88fb-f709-4c95-828d-9ef46ff8e159" containerName="console" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504099 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="prom-label-proxy" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504104 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="alertmanager" Nov 01 08:33:42.504441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.504109 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" containerName="kube-rbac-proxy" Nov 01 08:33:42.509007 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.508988 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.558318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558255 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 08:33:42.558318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558258 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 08:33:42.558318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558319 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 08:33:42.558318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558269 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 08:33:42.558707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558366 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 08:33:42.558707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558412 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 08:33:42.558707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558415 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 08:33:42.558707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558491 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7q4h\"" Nov 01 08:33:42.558707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.558548 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 08:33:42.605311 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.605276 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 08:33:42.615359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.615320 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:33:42.683568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683529 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.683568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683568 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-config-out\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.683841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683672 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5q8x\" (UniqueName: \"kubernetes.io/projected/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-kube-api-access-n5q8x\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.683841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683759 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-config-volume\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.683841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683789 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.683841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683811 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.683967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683896 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.684004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.683967 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.684036 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.684004 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.684068 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.684031 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-web-config\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.684100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.684071 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.684134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.684100 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.684163 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.684152 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785102 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785061 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-config-volume\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785102 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785100 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785284 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785118 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785284 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785243 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785411 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785292 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785435 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785482 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-web-config\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785552 5710 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 08:33:42.785617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785595 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785654 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785688 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785755 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785786 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-config-out\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.785937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.785846 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5q8x\" (UniqueName: \"kubernetes.io/projected/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-kube-api-access-n5q8x\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.786166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.786131 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.787925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.786605 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.787925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.786701 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.788909 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.788739 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-config-volume\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.788909 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.788831 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.788909 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.788831 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-web-config\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.789095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.788924 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.789095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.789042 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-config-out\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.789282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.789263 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.789429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.789410 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.789798 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.789778 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.790688 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.790667 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:42.843693 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:42.843662 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5q8x\" (UniqueName: \"kubernetes.io/projected/21a6f7a9-e5f9-4a81-9808-007cac6ccac0-kube-api-access-n5q8x\") pod \"alertmanager-main-0\" (UID: \"21a6f7a9-e5f9-4a81-9808-007cac6ccac0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:43.056012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:43.055933 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd38f917-f55b-458f-bfd4-18ea32a077af" path="/var/lib/kubelet/pods/dd38f917-f55b-458f-bfd4-18ea32a077af/volumes" Nov 01 08:33:43.133110 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:43.133071 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 08:33:43.618604 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:43.618571 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 08:33:43.627234 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:33:43.627198 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a6f7a9_e5f9_4a81_9808_007cac6ccac0.slice/crio-75b2b1cb10f22de3caecd456c5a61947d1bddf69c6f7ef3ad47def15eb8707c3 WatchSource:0}: Error finding container 75b2b1cb10f22de3caecd456c5a61947d1bddf69c6f7ef3ad47def15eb8707c3: Status 404 returned error can't find the container with id 75b2b1cb10f22de3caecd456c5a61947d1bddf69c6f7ef3ad47def15eb8707c3 Nov 01 08:33:44.311266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:44.311236 5710 generic.go:356] "Generic (PLEG): container finished" podID="21a6f7a9-e5f9-4a81-9808-007cac6ccac0" containerID="9fd17ab4bc0058c364beb2395bc980846a2aa046680db979290aab4f1407aaaf" exitCode=0 Nov 01 08:33:44.311460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:44.311328 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerDied","Data":"9fd17ab4bc0058c364beb2395bc980846a2aa046680db979290aab4f1407aaaf"} Nov 01 08:33:44.311460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:44.311404 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"75b2b1cb10f22de3caecd456c5a61947d1bddf69c6f7ef3ad47def15eb8707c3"} Nov 01 08:33:45.318645 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.318606 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"01e4ca498ee3330cc76ae700a9dfae8d61d068fdb43c711dd2de4b4cc59d6be9"} Nov 01 08:33:45.319200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.318673 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"8ab571bb0fcd9e3e79471624fd51f01a0e4d3440e3188538b0d151dd746ee998"} Nov 01 08:33:45.319200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.318700 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"bcd8396ad69acd9ec4487c54d076f3d29b9123e9c4bdaf4acb87cb099953e5f2"} Nov 01 08:33:45.319200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.318752 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"f7c9e174f251c43b33143047be871f03ac00ec21d360a0f3d01990d9537364f4"} Nov 01 08:33:45.319200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.318774 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"0c9ac6179ec6a7fae43726892dbec92263c14769e1c62637247481f4ea10da1e"} Nov 01 08:33:45.319200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.318797 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21a6f7a9-e5f9-4a81-9808-007cac6ccac0","Type":"ContainerStarted","Data":"9920c03d5669875226156f5de557d57b649b54cc97b1296df7325fca088c41a5"} Nov 01 08:33:45.357948 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.357907 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn"] Nov 01 08:33:45.361550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.361522 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.410640 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.410598 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 08:33:45.410908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.410649 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 08:33:45.410908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.410761 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 08:33:45.410908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.410813 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-76nvl\"" Nov 01 08:33:45.410908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.410836 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 08:33:45.410908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.410891 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 08:33:45.457626 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.457591 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 08:33:45.479849 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:33:45.479810 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:33:45.507151 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.507082 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.492991441 podStartE2EDuration="3.507065789s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.074348ms" isStatefulPod=false podCreationTimestamp="2026-11-01 08:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:33:45.458049662 +0000 UTC m=+48.546386334" watchObservedRunningTime="2026-11-01 08:33:45.507065789 +0000 UTC m=+48.595402462" Nov 01 08:33:45.507800 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.507697 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn"] Nov 01 08:33:45.509497 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.509473 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-federate-client-tls\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.509630 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.509509 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.509701 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.509662 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-secret-telemeter-client\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.509928 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.509904 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.510007 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.509981 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-metrics-client-ca\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.510076 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.510057 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspqm\" (UniqueName: \"kubernetes.io/projected/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-kube-api-access-mspqm\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.510240 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.510221 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-telemeter-client-tls\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.510302 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.510284 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-serving-certs-ca-bundle\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611261 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611306 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-metrics-client-ca\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611330 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspqm\" (UniqueName: \"kubernetes.io/projected/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-kube-api-access-mspqm\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611367 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-telemeter-client-tls\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611396 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-serving-certs-ca-bundle\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611413 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-federate-client-tls\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611436 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.611568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.611477 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-secret-telemeter-client\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.612304 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.612194 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-metrics-client-ca\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.612304 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.612194 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-serving-certs-ca-bundle\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.612986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.612958 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.614058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.614030 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.614368 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.614351 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-telemeter-client-tls\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.614752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.614711 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-secret-telemeter-client\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.614814 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.614710 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-federate-client-tls\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.669159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.669129 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspqm\" (UniqueName: \"kubernetes.io/projected/dfbd8ec8-8e1a-436c-b5cc-493861c60fe1-kube-api-access-mspqm\") pod \"telemeter-client-5bf5b6787f-ckmvn\" (UID: \"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1\") " pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:45.683972 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:45.683935 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" Nov 01 08:33:46.112057 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:46.112019 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn"] Nov 01 08:33:46.120171 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:33:46.120141 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbd8ec8_8e1a_436c_b5cc_493861c60fe1.slice/crio-488fe663d826c8254882bf590c919717c74fd810d29acdf8aeb55d6f0e6f1777 WatchSource:0}: Error finding container 488fe663d826c8254882bf590c919717c74fd810d29acdf8aeb55d6f0e6f1777: Status 404 returned error can't find the container with id 488fe663d826c8254882bf590c919717c74fd810d29acdf8aeb55d6f0e6f1777 Nov 01 08:33:46.122240 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:46.122225 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 08:33:46.324490 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:46.324445 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" event={"ID":"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1","Type":"ContainerStarted","Data":"488fe663d826c8254882bf590c919717c74fd810d29acdf8aeb55d6f0e6f1777"} Nov 01 08:33:49.341857 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.341821 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" event={"ID":"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1","Type":"ContainerStarted","Data":"4996e9f973c31ef49d8b2080ecb562e16e0d05257463e0fb95c1d838d6b82d76"} Nov 01 08:33:49.342259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.341882 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" event={"ID":"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1","Type":"ContainerStarted","Data":"26a9e6e7f86c220d9459c711eec62a87e6b517b55a85cc6b90ba303090fcfbd8"} Nov 01 08:33:49.342259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.341904 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" event={"ID":"dfbd8ec8-8e1a-436c-b5cc-493861c60fe1","Type":"ContainerStarted","Data":"ff3dfc3efd9461489176cc738274628821bf12e31eff79b908d0a6888f01c26f"} Nov 01 08:33:49.551348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.551279 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bf5b6787f-ckmvn" podStartSLOduration=2.196697747 podStartE2EDuration="4.551265044s" totalImagesPullingTime="2.354567297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:33:49.551249372 +0000 UTC m=+52.639586044" watchObservedRunningTime="2026-11-01 08:33:49.551265044 +0000 UTC m=+52.639601713" Nov 01 08:33:49.939641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.939606 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f5bc976cf-sllrk"] Nov 01 08:33:49.943125 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.943104 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:49.998508 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:49.998471 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5bc976cf-sllrk"] Nov 01 08:33:50.053944 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.053909 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.053944 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.053943 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.054187 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.053969 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.054187 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.053991 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cwh\" (UniqueName: \"kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.054187 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.054168 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.054329 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.054227 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.054329 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.054308 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.054399 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.054339 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155027 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.154977 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155027 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155025 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155164 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155212 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155252 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155274 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155507 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155390 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.155507 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155450 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cwh\" (UniqueName: \"kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.156030 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.155989 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.156154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.156032 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.156222 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.156157 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.156387 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.156354 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.156587 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.156564 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.158021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.157984 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.158091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.158045 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.213364 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.213282 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cwh\" (UniqueName: \"kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh\") pod \"console-5f5bc976cf-sllrk\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.255294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.255252 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:33:50.633317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:50.633281 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5bc976cf-sllrk"] Nov 01 08:33:50.642050 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:33:50.642021 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c53314_286d_4754_ab17_e7cd7a08ed12.slice/crio-98789727ef0f46b10a4c25b368157125df3b09de3c203740b1b0c95bb21612bb WatchSource:0}: Error finding container 98789727ef0f46b10a4c25b368157125df3b09de3c203740b1b0c95bb21612bb: Status 404 returned error can't find the container with id 98789727ef0f46b10a4c25b368157125df3b09de3c203740b1b0c95bb21612bb Nov 01 08:33:51.350810 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:51.350775 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5bc976cf-sllrk" event={"ID":"d6c53314-286d-4754-ab17-e7cd7a08ed12","Type":"ContainerStarted","Data":"6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480"} Nov 01 08:33:51.350987 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:51.350828 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5bc976cf-sllrk" event={"ID":"d6c53314-286d-4754-ab17-e7cd7a08ed12","Type":"ContainerStarted","Data":"98789727ef0f46b10a4c25b368157125df3b09de3c203740b1b0c95bb21612bb"} Nov 01 08:33:51.461853 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:33:51.461787 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f5bc976cf-sllrk" podStartSLOduration=2.46176697 podStartE2EDuration="2.46176697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:33:51.461479788 +0000 UTC m=+54.549816472" watchObservedRunningTime="2026-11-01 08:33:51.46176697 +0000 UTC m=+54.550103643" Nov 01 08:34:00.255700 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:00.255649 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:34:00.256142 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:00.255746 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:34:00.261159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:00.261130 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:34:00.388545 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:00.388519 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:34:00.532882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:00.532788 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6958777449-f4bt5"] Nov 01 08:34:15.475560 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:34:15.475524 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:34:25.553353 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.553290 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6958777449-f4bt5" podUID="17bada36-7447-4f33-9279-ef2b7c1a0c33" containerName="console" containerID="cri-o://656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b" gracePeriod=15 Nov 01 08:34:25.815701 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.815638 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:34:25.971870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.971834 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.971870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.971885 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.971911 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.971933 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.971964 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.971995 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972024 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972103 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") pod \"17bada36-7447-4f33-9279-ef2b7c1a0c33\" (UID: \"17bada36-7447-4f33-9279-ef2b7c1a0c33\") " Nov 01 08:34:25.972425 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972266 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:34:25.972482 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972439 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:34:25.972535 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972492 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:34:25.972591 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972560 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:34:25.972779 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.972690 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:34:25.974570 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.974545 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:34:25.974682 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.974636 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "kube-api-access-66p5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:34:25.974850 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:25.974806 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert" pod "17bada36-7447-4f33-9279-ef2b7c1a0c33" (UID: "17bada36-7447-4f33-9279-ef2b7c1a0c33"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073578 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66p5s\" (UniqueName: \"kubernetes.io/projected/17bada36-7447-4f33-9279-ef2b7c1a0c33-kube-api-access-66p5s\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073614 5710 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073631 5710 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-oauth-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073645 5710 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bada36-7447-4f33-9279-ef2b7c1a0c33-tmp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073658 5710 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073670 5710 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-oauth-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.073680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073683 5710 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bada36-7447-4f33-9279-ef2b7c1a0c33-trusted-ca-bundle\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.074056 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.073696 5710 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17bada36-7447-4f33-9279-ef2b7c1a0c33-console-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:34:26.476127 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.476088 5710 generic.go:356] "Generic (PLEG): container finished" podID="17bada36-7447-4f33-9279-ef2b7c1a0c33" containerID="656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b" exitCode=0 Nov 01 08:34:26.476310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.476141 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6958777449-f4bt5" Nov 01 08:34:26.476310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.476136 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6958777449-f4bt5" event={"ID":"17bada36-7447-4f33-9279-ef2b7c1a0c33","Type":"ContainerDied","Data":"656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b"} Nov 01 08:34:26.476310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.476195 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6958777449-f4bt5" event={"ID":"17bada36-7447-4f33-9279-ef2b7c1a0c33","Type":"ContainerDied","Data":"96d3c09554f8bbc57cc70a67593ebe91a64a34bbaa89c4bb2210a20eed65e473"} Nov 01 08:34:26.476310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.476216 5710 scope.go:122] "RemoveContainer" containerID="656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b" Nov 01 08:34:26.484940 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.484925 5710 scope.go:122] "RemoveContainer" containerID="656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b" Nov 01 08:34:26.485197 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:34:26.485171 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b\": container with ID starting with 656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b not found: ID does not exist" containerID="656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b" Nov 01 08:34:26.485259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.485205 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b"} err="failed to get container status \"656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b\": rpc error: code = NotFound desc = could not find container \"656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b\": container with ID starting with 656aa4e69447278ff2c491129279de68baa22a240c42c417adab5c517c5b515b not found: ID does not exist" Nov 01 08:34:26.629356 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.629324 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6958777449-f4bt5"] Nov 01 08:34:26.675642 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:26.675608 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6958777449-f4bt5"] Nov 01 08:34:27.055761 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:34:27.055703 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bada36-7447-4f33-9279-ef2b7c1a0c33" path="/var/lib/kubelet/pods/17bada36-7447-4f33-9279-ef2b7c1a0c33/volumes" Nov 01 08:34:45.476071 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:34:45.475983 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:35:15.475709 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:35:15.475667 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:35:23.858653 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.858614 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-595b877ff6-p6lpn"] Nov 01 08:35:23.859224 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.858925 5710 cpu_manager.go:414] "containerMap: removing container" podUID="17bada36-7447-4f33-9279-ef2b7c1a0c33" containerName="console" Nov 01 08:35:23.859224 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.858937 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bada36-7447-4f33-9279-ef2b7c1a0c33" containerName="console" Nov 01 08:35:23.859224 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.859011 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bada36-7447-4f33-9279-ef2b7c1a0c33" containerName="console" Nov 01 08:35:23.864249 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.863002 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.938627 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.938582 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-oauth-serving-cert\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.938844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.938644 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-serving-cert\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.938844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.938756 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-service-ca\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.938949 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.938857 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-oauth-config\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.938949 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.938900 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a58d852-ca9c-468a-bc3b-f64a991f7afa-tmp\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.939027 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.939001 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-config\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.939078 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.939039 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-trusted-ca-bundle\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.939078 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.939063 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8qr\" (UniqueName: \"kubernetes.io/projected/7a58d852-ca9c-468a-bc3b-f64a991f7afa-kube-api-access-lz8qr\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:23.954676 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:23.954644 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595b877ff6-p6lpn"] Nov 01 08:35:24.040143 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040095 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-serving-cert\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040143 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040141 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-service-ca\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040166 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-oauth-config\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040191 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a58d852-ca9c-468a-bc3b-f64a991f7afa-tmp\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040226 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-config\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040244 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-trusted-ca-bundle\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040266 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8qr\" (UniqueName: \"kubernetes.io/projected/7a58d852-ca9c-468a-bc3b-f64a991f7afa-kube-api-access-lz8qr\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.040424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040296 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-oauth-serving-cert\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.041032 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.040995 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-service-ca\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.041300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.041029 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-oauth-serving-cert\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.041300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.041116 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a58d852-ca9c-468a-bc3b-f64a991f7afa-tmp\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.041423 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.041368 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-config\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.041967 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.041940 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a58d852-ca9c-468a-bc3b-f64a991f7afa-trusted-ca-bundle\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.043048 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.043030 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-serving-cert\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.043165 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.043143 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a58d852-ca9c-468a-bc3b-f64a991f7afa-console-oauth-config\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.099157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.099118 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8qr\" (UniqueName: \"kubernetes.io/projected/7a58d852-ca9c-468a-bc3b-f64a991f7afa-kube-api-access-lz8qr\") pod \"console-595b877ff6-p6lpn\" (UID: \"7a58d852-ca9c-468a-bc3b-f64a991f7afa\") " pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.175751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.175643 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:24.549908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.549826 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595b877ff6-p6lpn"] Nov 01 08:35:24.559513 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:35:24.559487 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a58d852_ca9c_468a_bc3b_f64a991f7afa.slice/crio-85eab6f69210207818d3c2388bb2bcd19ff86f1a232b8904a5cc60b3315d26a4 WatchSource:0}: Error finding container 85eab6f69210207818d3c2388bb2bcd19ff86f1a232b8904a5cc60b3315d26a4: Status 404 returned error can't find the container with id 85eab6f69210207818d3c2388bb2bcd19ff86f1a232b8904a5cc60b3315d26a4 Nov 01 08:35:24.694022 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.693987 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595b877ff6-p6lpn" event={"ID":"7a58d852-ca9c-468a-bc3b-f64a991f7afa","Type":"ContainerStarted","Data":"4e0aaf56e93a01985e710521d6d5ca0e667f4dec018fc1fed07fd579b9a47bb7"} Nov 01 08:35:24.694194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.694043 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595b877ff6-p6lpn" event={"ID":"7a58d852-ca9c-468a-bc3b-f64a991f7afa","Type":"ContainerStarted","Data":"85eab6f69210207818d3c2388bb2bcd19ff86f1a232b8904a5cc60b3315d26a4"} Nov 01 08:35:24.804972 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:24.804859 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-595b877ff6-p6lpn" podStartSLOduration=1.804843811 podStartE2EDuration="1.804843811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:35:24.80469278 +0000 UTC m=+147.893029453" watchObservedRunningTime="2026-11-01 08:35:24.804843811 +0000 UTC m=+147.893180480" Nov 01 08:35:34.176442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:34.176397 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:34.176442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:34.176454 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:34.181420 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:34.181394 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:34.734793 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:34.734762 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595b877ff6-p6lpn" Nov 01 08:35:34.874738 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:34.874687 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5bc976cf-sllrk"] Nov 01 08:35:45.477710 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:35:45.477672 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:35:59.895857 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:35:59.895796 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f5bc976cf-sllrk" podUID="d6c53314-286d-4754-ab17-e7cd7a08ed12" containerName="console" containerID="cri-o://6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480" gracePeriod=15 Nov 01 08:36:00.144662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.144626 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:36:00.238466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238360 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238403 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238431 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238454 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238848 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238502 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238848 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238520 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238848 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238555 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh\" (UniqueName: \"kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.238848 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238708 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp\") pod \"d6c53314-286d-4754-ab17-e7cd7a08ed12\" (UID: \"d6c53314-286d-4754-ab17-e7cd7a08ed12\") " Nov 01 08:36:00.239037 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238955 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:36:00.239037 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.238973 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:36:00.239235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.239208 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:36:00.239358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.239212 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:36:00.239574 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.239545 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:36:00.240996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.240932 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:36:00.240996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.240954 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "kube-api-access-22cwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:36:00.240996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.240979 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config" pod "d6c53314-286d-4754-ab17-e7cd7a08ed12" (UID: "d6c53314-286d-4754-ab17-e7cd7a08ed12"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:36:00.339623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339582 5710 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-trusted-ca-bundle\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339628 5710 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-oauth-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339639 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cwh\" (UniqueName: \"kubernetes.io/projected/d6c53314-286d-4754-ab17-e7cd7a08ed12-kube-api-access-22cwh\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339650 5710 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6c53314-286d-4754-ab17-e7cd7a08ed12-tmp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339662 5710 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-oauth-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339670 5710 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-config\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339679 5710 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6c53314-286d-4754-ab17-e7cd7a08ed12-console-serving-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.339888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.339687 5710 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c53314-286d-4754-ab17-e7cd7a08ed12-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:36:00.823912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.823871 5710 generic.go:356] "Generic (PLEG): container finished" podID="d6c53314-286d-4754-ab17-e7cd7a08ed12" containerID="6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480" exitCode=0 Nov 01 08:36:00.824134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.823929 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5bc976cf-sllrk" Nov 01 08:36:00.824134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.823958 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5bc976cf-sllrk" event={"ID":"d6c53314-286d-4754-ab17-e7cd7a08ed12","Type":"ContainerDied","Data":"6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480"} Nov 01 08:36:00.824134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.824017 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5bc976cf-sllrk" event={"ID":"d6c53314-286d-4754-ab17-e7cd7a08ed12","Type":"ContainerDied","Data":"98789727ef0f46b10a4c25b368157125df3b09de3c203740b1b0c95bb21612bb"} Nov 01 08:36:00.824134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.824039 5710 scope.go:122] "RemoveContainer" containerID="6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480" Nov 01 08:36:00.833192 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.833173 5710 scope.go:122] "RemoveContainer" containerID="6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480" Nov 01 08:36:00.833496 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:36:00.833481 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480\": container with ID starting with 6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480 not found: ID does not exist" containerID="6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480" Nov 01 08:36:00.833538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.833508 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480"} err="failed to get container status \"6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480\": rpc error: code = NotFound desc = could not find container \"6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480\": container with ID starting with 6917dac36df079ae7b61e8c250f9469fbe6e4f42b7b54f3a032339a0906d3480 not found: ID does not exist" Nov 01 08:36:00.979420 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:00.979381 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5bc976cf-sllrk"] Nov 01 08:36:01.025876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:01.025839 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f5bc976cf-sllrk"] Nov 01 08:36:01.055911 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:36:01.055877 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c53314-286d-4754-ab17-e7cd7a08ed12" path="/var/lib/kubelet/pods/d6c53314-286d-4754-ab17-e7cd7a08ed12/volumes" Nov 01 08:36:15.476643 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:36:15.476597 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:36:45.476010 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:36:45.475972 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:37:09.752058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.752014 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp"] Nov 01 08:37:09.752626 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.752368 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d6c53314-286d-4754-ab17-e7cd7a08ed12" containerName="console" Nov 01 08:37:09.752626 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.752379 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c53314-286d-4754-ab17-e7cd7a08ed12" containerName="console" Nov 01 08:37:09.752626 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.752466 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c53314-286d-4754-ab17-e7cd7a08ed12" containerName="console" Nov 01 08:37:09.755593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.755576 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.804339 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.804304 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 08:37:09.850755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.850701 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-db9jj\"" Nov 01 08:37:09.850755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.850701 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 08:37:09.851462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.851440 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp"] Nov 01 08:37:09.893080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.893047 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.893223 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.893082 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.893223 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.893105 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vbl\" (UniqueName: \"kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.993657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.993609 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.993657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.993653 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.993964 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.993677 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vbl\" (UniqueName: \"kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.994069 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.994047 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:09.994133 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:09.994114 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:10.052791 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.052685 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vbl\" (UniqueName: \"kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:10.074899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.074850 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:10.291429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.291399 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9"] Nov 01 08:37:10.294925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.294904 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.345394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.345365 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 08:37:10.345394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.345384 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 08:37:10.351457 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.351427 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 08:37:10.351616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.351465 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 08:37:10.351616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.351484 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 08:37:10.351616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.351498 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 08:37:10.351616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.351540 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z5ppc\"" Nov 01 08:37:10.360583 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.360549 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9"] Nov 01 08:37:10.397907 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.397873 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.398099 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.397928 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.398099 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.398038 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.398099 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.398077 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.398259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.398099 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65cw\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.398316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.398250 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.398316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.398295 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.476164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.476133 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp"] Nov 01 08:37:10.485271 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:37:10.485238 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2597e0_5997_4bf9_bb8d_6a40b2b666c3.slice/crio-1a75e7fdf71c9e035b5ed27317e78b244582a6aa7a2d4602878ecd911d189e8e WatchSource:0}: Error finding container 1a75e7fdf71c9e035b5ed27317e78b244582a6aa7a2d4602878ecd911d189e8e: Status 404 returned error can't find the container with id 1a75e7fdf71c9e035b5ed27317e78b244582a6aa7a2d4602878ecd911d189e8e Nov 01 08:37:10.499519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499489 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.499659 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499529 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.499659 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499561 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.499659 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499587 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65cw\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.499659 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499644 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.499925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499673 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.499925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.499710 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.500330 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.500307 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.502082 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.502061 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.502194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.502114 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.502492 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.502472 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.502574 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.502475 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.557298 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.557262 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.557454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.557308 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65cw\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw\") pod \"istiod-openshift-gateway-7876bfcb74-swgd9\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:10.607749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:10.607628 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:11.033261 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:11.033164 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9"] Nov 01 08:37:11.041559 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:37:11.041530 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c43382d_ec74_4862_b34e_76efe2587f37.slice/crio-f94e0d37718acdff7604f0e1758146845c5645f288754e31f0de2bbec7b14695 WatchSource:0}: Error finding container f94e0d37718acdff7604f0e1758146845c5645f288754e31f0de2bbec7b14695: Status 404 returned error can't find the container with id f94e0d37718acdff7604f0e1758146845c5645f288754e31f0de2bbec7b14695 Nov 01 08:37:11.082676 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:11.082638 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" event={"ID":"2c43382d-ec74-4862-b34e-76efe2587f37","Type":"ContainerStarted","Data":"f94e0d37718acdff7604f0e1758146845c5645f288754e31f0de2bbec7b14695"} Nov 01 08:37:11.083829 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:11.083805 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" event={"ID":"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3","Type":"ContainerStarted","Data":"1a75e7fdf71c9e035b5ed27317e78b244582a6aa7a2d4602878ecd911d189e8e"} Nov 01 08:37:15.428706 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:15.428670 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:37:15.428706 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:15.428767 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:37:15.493133 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:37:15.493097 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:37:16.116293 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:16.116240 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" event={"ID":"2c43382d-ec74-4862-b34e-76efe2587f37","Type":"ContainerStarted","Data":"963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d"} Nov 01 08:37:16.116538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:16.116484 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:16.118378 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:16.118350 5710 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-swgd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 08:37:16.118531 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:16.118408 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 08:37:16.309784 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:16.309694 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" podStartSLOduration=1.925837799 podStartE2EDuration="6.309677262s" totalImagesPullingTime="4.383839463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:16.30898168 +0000 UTC m=+259.397318352" watchObservedRunningTime="2026-11-01 08:37:16.309677262 +0000 UTC m=+259.398013931" Nov 01 08:37:17.122237 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:17.122205 5710 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-swgd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 08:37:17.122664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:17.122267 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 08:37:20.122763 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:20.122732 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:37:21.139647 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:21.139608 5710 generic.go:356] "Generic (PLEG): container finished" podID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerID="bba04bcfac202102b8ccd37818d70a6a457f5ccc02af525cd3bf32e76ec9b617" exitCode=0 Nov 01 08:37:21.140126 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:21.139700 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" event={"ID":"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3","Type":"ContainerDied","Data":"bba04bcfac202102b8ccd37818d70a6a457f5ccc02af525cd3bf32e76ec9b617"} Nov 01 08:37:23.149282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:23.149188 5710 generic.go:356] "Generic (PLEG): container finished" podID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerID="e1ce7d46b428528ced2c20b32ae94e3577236bf15051699a493ceccef729f462" exitCode=0 Nov 01 08:37:23.149282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:23.149226 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" event={"ID":"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3","Type":"ContainerDied","Data":"e1ce7d46b428528ced2c20b32ae94e3577236bf15051699a493ceccef729f462"} Nov 01 08:37:30.180627 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:30.180588 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" event={"ID":"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3","Type":"ContainerStarted","Data":"e6304bb349b6c61fe1860dece14a2576583224aa9b943a7a9cecece20b23b80b"} Nov 01 08:37:30.290844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:30.290735 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" podStartSLOduration=3.033874957 podStartE2EDuration="21.290697188s" totalImagesPullingTime="18.256822231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:37:30.290372608 +0000 UTC m=+273.378709284" watchObservedRunningTime="2026-11-01 08:37:30.290697188 +0000 UTC m=+273.379033937" Nov 01 08:37:31.186260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:31.186223 5710 generic.go:356] "Generic (PLEG): container finished" podID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerID="e6304bb349b6c61fe1860dece14a2576583224aa9b943a7a9cecece20b23b80b" exitCode=0 Nov 01 08:37:31.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:31.186271 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" event={"ID":"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3","Type":"ContainerDied","Data":"e6304bb349b6c61fe1860dece14a2576583224aa9b943a7a9cecece20b23b80b"} Nov 01 08:37:32.320929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.320900 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:32.385681 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.385648 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util\") pod \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " Nov 01 08:37:32.385902 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.385728 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle\") pod \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " Nov 01 08:37:32.385902 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.385778 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl\" (UniqueName: \"kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl\") pod \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\" (UID: \"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3\") " Nov 01 08:37:32.386242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.386216 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle" pod "cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" (UID: "cf2597e0-5997-4bf9-bb8d-6a40b2b666c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:37:32.388137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.388107 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl" pod "cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" (UID: "cf2597e0-5997-4bf9-bb8d-6a40b2b666c3"). InnerVolumeSpecName "kube-api-access-p5vbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:37:32.390887 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.390864 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util" pod "cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" (UID: "cf2597e0-5997-4bf9-bb8d-6a40b2b666c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:37:32.486662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.486582 5710 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-bundle\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:37:32.486662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.486609 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vbl\" (UniqueName: \"kubernetes.io/projected/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-kube-api-access-p5vbl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:37:32.486662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:32.486618 5710 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2597e0-5997-4bf9-bb8d-6a40b2b666c3-util\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:37:33.195105 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:33.195067 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" event={"ID":"cf2597e0-5997-4bf9-bb8d-6a40b2b666c3","Type":"ContainerDied","Data":"1a75e7fdf71c9e035b5ed27317e78b244582a6aa7a2d4602878ecd911d189e8e"} Nov 01 08:37:33.195105 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:33.195086 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71826tp" Nov 01 08:37:33.195344 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:33.195125 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a75e7fdf71c9e035b5ed27317e78b244582a6aa7a2d4602878ecd911d189e8e" Nov 01 08:37:45.476326 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:37:45.476244 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:37:54.729557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.729511 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z"] Nov 01 08:37:54.730063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730039 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="pull" Nov 01 08:37:54.730063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730059 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="pull" Nov 01 08:37:54.730229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730082 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="extract" Nov 01 08:37:54.730229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730088 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="extract" Nov 01 08:37:54.730229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730115 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="util" Nov 01 08:37:54.730229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730122 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="util" Nov 01 08:37:54.730229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.730204 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2597e0-5997-4bf9-bb8d-6a40b2b666c3" containerName="extract" Nov 01 08:37:54.733657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.733628 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.782350 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.782313 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 08:37:54.782613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.782414 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 08:37:54.782613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.782469 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 08:37:54.782613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.782482 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 08:37:54.782613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.782500 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 08:37:54.782613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.782568 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gx5kp\"" Nov 01 08:37:54.857299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.857264 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21662f25-78a3-40c5-9940-503313b5ea9b-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.857511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.857318 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.857511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.857411 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21662f25-78a3-40c5-9940-503313b5ea9b-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.857633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.857506 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.857633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.857552 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.857785 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.857630 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqp8q\" (UniqueName: \"kubernetes.io/projected/21662f25-78a3-40c5-9940-503313b5ea9b-kube-api-access-dqp8q\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.864317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.864284 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z"] Nov 01 08:37:54.958552 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.958513 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqp8q\" (UniqueName: \"kubernetes.io/projected/21662f25-78a3-40c5-9940-503313b5ea9b-kube-api-access-dqp8q\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.958791 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.958564 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21662f25-78a3-40c5-9940-503313b5ea9b-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.958791 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.958595 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.958791 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.958611 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21662f25-78a3-40c5-9940-503313b5ea9b-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.958791 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.958653 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.958791 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.958697 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.959107 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.959079 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21662f25-78a3-40c5-9940-503313b5ea9b-manifests\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.961496 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.961466 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-metrics-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.961606 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.961503 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.961741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.961698 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21662f25-78a3-40c5-9940-503313b5ea9b-tmp\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:54.961966 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:54.961947 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21662f25-78a3-40c5-9940-503313b5ea9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:55.016365 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:55.016297 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqp8q\" (UniqueName: \"kubernetes.io/projected/21662f25-78a3-40c5-9940-503313b5ea9b-kube-api-access-dqp8q\") pod \"opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z\" (UID: \"21662f25-78a3-40c5-9940-503313b5ea9b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:55.049308 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:55.049270 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:37:55.540977 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:55.540942 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z"] Nov 01 08:37:55.549404 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:37:55.549376 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21662f25_78a3_40c5_9940_503313b5ea9b.slice/crio-2b1426ec9a9242e4bd04a3f7873b23a531dbffd73a6c7061eb16ae84aae8bae7 WatchSource:0}: Error finding container 2b1426ec9a9242e4bd04a3f7873b23a531dbffd73a6c7061eb16ae84aae8bae7: Status 404 returned error can't find the container with id 2b1426ec9a9242e4bd04a3f7873b23a531dbffd73a6c7061eb16ae84aae8bae7 Nov 01 08:37:56.286118 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:56.286081 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" event={"ID":"21662f25-78a3-40c5-9940-503313b5ea9b","Type":"ContainerStarted","Data":"2b1426ec9a9242e4bd04a3f7873b23a531dbffd73a6c7061eb16ae84aae8bae7"} Nov 01 08:37:57.132683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:57.132602 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:37:57.132918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:37:57.132786 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:38:00.307082 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:00.307038 5710 generic.go:356] "Generic (PLEG): container finished" podID="21662f25-78a3-40c5-9940-503313b5ea9b" containerID="cc024d18d0681c196bd3fb95584ee35e5681167e941a2aeccbb57434b04fef13" exitCode=0 Nov 01 08:38:00.307488 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:00.307098 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" event={"ID":"21662f25-78a3-40c5-9940-503313b5ea9b","Type":"ContainerDied","Data":"cc024d18d0681c196bd3fb95584ee35e5681167e941a2aeccbb57434b04fef13"} Nov 01 08:38:02.318949 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:02.318912 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" event={"ID":"21662f25-78a3-40c5-9940-503313b5ea9b","Type":"ContainerStarted","Data":"443452623ffb95af92c923793d1b1b2ea5ff0b2a49aee09f3d10d218e2ee8bfd"} Nov 01 08:38:02.319517 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:02.319051 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:38:02.591204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:02.591121 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" podStartSLOduration=3.088285253 podStartE2EDuration="8.591102131s" totalImagesPullingTime="5.117187207s" totalInitContainerRuntime="385.629671ms" isStatefulPod=false podCreationTimestamp="2026-11-01 08:37:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:02.543824805 +0000 UTC m=+305.632161503" watchObservedRunningTime="2026-11-01 08:38:02.591102131 +0000 UTC m=+305.679438801" Nov 01 08:38:13.325357 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:13.325327 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57cd9cc6fc-9vg7z" Nov 01 08:38:15.475910 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:38:15.475873 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:38:33.432974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.432941 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5649b5c55d-f88db"] Nov 01 08:38:33.502413 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.502376 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.515635 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.515587 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5649b5c55d-f88db"] Nov 01 08:38:33.587541 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.587500 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 08:38:33.587757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.587520 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rpqnp\"" Nov 01 08:38:33.587757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.587542 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 08:38:33.601562 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.601527 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t657j\" (UniqueName: \"kubernetes.io/projected/f1e8428b-1e87-425d-91e9-02dcc567fcb6-kube-api-access-t657j\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.601667 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.601574 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8428b-1e87-425d-91e9-02dcc567fcb6-tls-certs\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.601769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.601747 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e8428b-1e87-425d-91e9-02dcc567fcb6-tmp\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.702942 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.702855 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8428b-1e87-425d-91e9-02dcc567fcb6-tls-certs\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.702942 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.702923 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e8428b-1e87-425d-91e9-02dcc567fcb6-tmp\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.703148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.702963 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t657j\" (UniqueName: \"kubernetes.io/projected/f1e8428b-1e87-425d-91e9-02dcc567fcb6-kube-api-access-t657j\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.705406 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.705373 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1e8428b-1e87-425d-91e9-02dcc567fcb6-tmp\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.705555 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.705539 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8428b-1e87-425d-91e9-02dcc567fcb6-tls-certs\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.774005 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.773964 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t657j\" (UniqueName: \"kubernetes.io/projected/f1e8428b-1e87-425d-91e9-02dcc567fcb6-kube-api-access-t657j\") pod \"kube-auth-proxy-5649b5c55d-f88db\" (UID: \"f1e8428b-1e87-425d-91e9-02dcc567fcb6\") " pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:33.813521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:33.813486 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" Nov 01 08:38:34.194151 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:34.194109 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5649b5c55d-f88db"] Nov 01 08:38:34.202515 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:38:34.202468 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e8428b_1e87_425d_91e9_02dcc567fcb6.slice/crio-18dadc8ad0d4347d3dd47afaf191fc3fd08c0ee38fbda5f2277644e8e709e353 WatchSource:0}: Error finding container 18dadc8ad0d4347d3dd47afaf191fc3fd08c0ee38fbda5f2277644e8e709e353: Status 404 returned error can't find the container with id 18dadc8ad0d4347d3dd47afaf191fc3fd08c0ee38fbda5f2277644e8e709e353 Nov 01 08:38:34.453866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:34.453768 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" event={"ID":"f1e8428b-1e87-425d-91e9-02dcc567fcb6","Type":"ContainerStarted","Data":"18dadc8ad0d4347d3dd47afaf191fc3fd08c0ee38fbda5f2277644e8e709e353"} Nov 01 08:38:36.979566 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:36.979532 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-lx2wp"] Nov 01 08:38:36.983607 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:36.983568 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.032665 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.032626 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b84285-aafd-4520-add8-c0314083c763-bound-sa-token\") pod \"cert-manager-597b95cb84-lx2wp\" (UID: \"02b84285-aafd-4520-add8-c0314083c763\") " pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.032869 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.032785 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8wjz\" (UniqueName: \"kubernetes.io/projected/02b84285-aafd-4520-add8-c0314083c763-kube-api-access-s8wjz\") pod \"cert-manager-597b95cb84-lx2wp\" (UID: \"02b84285-aafd-4520-add8-c0314083c763\") " pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.035407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.035376 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-lx2wp"] Nov 01 08:38:37.080322 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.080262 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 08:38:37.080322 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.080277 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 08:38:37.084297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.084248 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mkhtc\"" Nov 01 08:38:37.134088 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.134051 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8wjz\" (UniqueName: \"kubernetes.io/projected/02b84285-aafd-4520-add8-c0314083c763-kube-api-access-s8wjz\") pod \"cert-manager-597b95cb84-lx2wp\" (UID: \"02b84285-aafd-4520-add8-c0314083c763\") " pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.134290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.134138 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b84285-aafd-4520-add8-c0314083c763-bound-sa-token\") pod \"cert-manager-597b95cb84-lx2wp\" (UID: \"02b84285-aafd-4520-add8-c0314083c763\") " pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.193808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.193773 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b84285-aafd-4520-add8-c0314083c763-bound-sa-token\") pod \"cert-manager-597b95cb84-lx2wp\" (UID: \"02b84285-aafd-4520-add8-c0314083c763\") " pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.193950 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.193847 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8wjz\" (UniqueName: \"kubernetes.io/projected/02b84285-aafd-4520-add8-c0314083c763-kube-api-access-s8wjz\") pod \"cert-manager-597b95cb84-lx2wp\" (UID: \"02b84285-aafd-4520-add8-c0314083c763\") " pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.314757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.314705 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-lx2wp" Nov 01 08:38:37.469339 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.469304 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" event={"ID":"f1e8428b-1e87-425d-91e9-02dcc567fcb6","Type":"ContainerStarted","Data":"c5d5863e2dc5551cdc10681e08f401fdd76fba240265fc12d3dcaa67543a3805"} Nov 01 08:38:37.578279 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.577995 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5649b5c55d-f88db" podStartSLOduration=1.51864946 podStartE2EDuration="4.577974581s" totalImagesPullingTime="3.059325121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:37.577705699 +0000 UTC m=+340.666042372" watchObservedRunningTime="2026-11-01 08:38:37.577974581 +0000 UTC m=+340.666311255" Nov 01 08:38:37.693644 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:37.693600 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-lx2wp"] Nov 01 08:38:37.702067 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:38:37.702036 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b84285_aafd_4520_add8_c0314083c763.slice/crio-f9eea75d7c64843b1137e39f5cafac55c96ea189ddea79218d5b5194462c22e9 WatchSource:0}: Error finding container f9eea75d7c64843b1137e39f5cafac55c96ea189ddea79218d5b5194462c22e9: Status 404 returned error can't find the container with id f9eea75d7c64843b1137e39f5cafac55c96ea189ddea79218d5b5194462c22e9 Nov 01 08:38:38.475302 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:38.475264 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-lx2wp" event={"ID":"02b84285-aafd-4520-add8-c0314083c763","Type":"ContainerStarted","Data":"f9eea75d7c64843b1137e39f5cafac55c96ea189ddea79218d5b5194462c22e9"} Nov 01 08:38:42.495647 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:42.495546 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-lx2wp" event={"ID":"02b84285-aafd-4520-add8-c0314083c763","Type":"ContainerStarted","Data":"65199d4033222a7370afb69de2cc6218fd18c78488d16716c58fd4f2241737a0"} Nov 01 08:38:42.603637 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:42.603585 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-lx2wp" podStartSLOduration=2.081170311 podStartE2EDuration="6.603569067s" totalImagesPullingTime="4.522398756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:38:42.603445582 +0000 UTC m=+345.691782254" watchObservedRunningTime="2026-11-01 08:38:42.603569067 +0000 UTC m=+345.691905739" Nov 01 08:38:45.478222 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:38:45.478186 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:38:59.902768 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:59.902728 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kmlrt"] Nov 01 08:38:59.907516 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:59.907478 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:38:59.958065 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:59.958031 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 08:38:59.959473 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:38:59.959439 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kmlrt"] Nov 01 08:39:00.004048 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.004011 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wlrzh\"" Nov 01 08:39:00.004048 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.004011 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 08:39:00.083194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.081168 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszx7\" (UniqueName: \"kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7\") pod \"authorino-56c6c9ddd5-kmlrt\" (UID: \"81fe0ef3-04a5-45f1-a67c-204ff2bd3293\") " pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:39:00.182602 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.182500 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszx7\" (UniqueName: \"kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7\") pod \"authorino-56c6c9ddd5-kmlrt\" (UID: \"81fe0ef3-04a5-45f1-a67c-204ff2bd3293\") " pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:39:00.240769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.240707 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszx7\" (UniqueName: \"kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7\") pod \"authorino-56c6c9ddd5-kmlrt\" (UID: \"81fe0ef3-04a5-45f1-a67c-204ff2bd3293\") " pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:39:00.521499 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.521405 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:39:00.891389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.891354 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kmlrt"] Nov 01 08:39:00.900406 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:39:00.900366 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81fe0ef3_04a5_45f1_a67c_204ff2bd3293.slice/crio-5f77ee1e81d2d9bb871a753b008b73e9e3de5d999a0e7fe9668f8ef9a1f93456 WatchSource:0}: Error finding container 5f77ee1e81d2d9bb871a753b008b73e9e3de5d999a0e7fe9668f8ef9a1f93456: Status 404 returned error can't find the container with id 5f77ee1e81d2d9bb871a753b008b73e9e3de5d999a0e7fe9668f8ef9a1f93456 Nov 01 08:39:00.902037 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:00.902013 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 08:39:01.583456 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:01.583416 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" event={"ID":"81fe0ef3-04a5-45f1-a67c-204ff2bd3293","Type":"ContainerStarted","Data":"5f77ee1e81d2d9bb871a753b008b73e9e3de5d999a0e7fe9668f8ef9a1f93456"} Nov 01 08:39:02.975849 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:02.975809 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:39:02.980679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:02.980597 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.080181 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.078674 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wx6gn\"" Nov 01 08:39:03.080181 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.078742 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 08:39:03.080181 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.080101 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:39:03.107454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.107410 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2cw\" (UniqueName: \"kubernetes.io/projected/308c91a4-a197-4420-8d55-7d7e9fe1e44d-kube-api-access-4w2cw\") pod \"limitador-limitador-7c9b87d574-2rfzc\" (UID: \"308c91a4-a197-4420-8d55-7d7e9fe1e44d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.107657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.107476 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/308c91a4-a197-4420-8d55-7d7e9fe1e44d-config-file\") pod \"limitador-limitador-7c9b87d574-2rfzc\" (UID: \"308c91a4-a197-4420-8d55-7d7e9fe1e44d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.208112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.208076 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/308c91a4-a197-4420-8d55-7d7e9fe1e44d-config-file\") pod \"limitador-limitador-7c9b87d574-2rfzc\" (UID: \"308c91a4-a197-4420-8d55-7d7e9fe1e44d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.208331 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.208239 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2cw\" (UniqueName: \"kubernetes.io/projected/308c91a4-a197-4420-8d55-7d7e9fe1e44d-kube-api-access-4w2cw\") pod \"limitador-limitador-7c9b87d574-2rfzc\" (UID: \"308c91a4-a197-4420-8d55-7d7e9fe1e44d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.209339 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.209287 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/308c91a4-a197-4420-8d55-7d7e9fe1e44d-config-file\") pod \"limitador-limitador-7c9b87d574-2rfzc\" (UID: \"308c91a4-a197-4420-8d55-7d7e9fe1e44d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.269368 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.269209 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2cw\" (UniqueName: \"kubernetes.io/projected/308c91a4-a197-4420-8d55-7d7e9fe1e44d-kube-api-access-4w2cw\") pod \"limitador-limitador-7c9b87d574-2rfzc\" (UID: \"308c91a4-a197-4420-8d55-7d7e9fe1e44d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.277416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.277381 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:39:03.298943 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.298900 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:03.691660 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:03.691625 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:39:03.700623 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:39:03.700580 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308c91a4_a197_4420_8d55_7d7e9fe1e44d.slice/crio-d32051db7a31554932058dcf80f6700bf737b326ced464c283e75f9bd2851170 WatchSource:0}: Error finding container d32051db7a31554932058dcf80f6700bf737b326ced464c283e75f9bd2851170: Status 404 returned error can't find the container with id d32051db7a31554932058dcf80f6700bf737b326ced464c283e75f9bd2851170 Nov 01 08:39:04.598572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:04.598483 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" event={"ID":"308c91a4-a197-4420-8d55-7d7e9fe1e44d","Type":"ContainerStarted","Data":"d32051db7a31554932058dcf80f6700bf737b326ced464c283e75f9bd2851170"} Nov 01 08:39:05.604466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:05.604426 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" event={"ID":"81fe0ef3-04a5-45f1-a67c-204ff2bd3293","Type":"ContainerStarted","Data":"ae543a9edab820052109380875fbdb1d85a3475b91e3702b5558f52f460e0637"} Nov 01 08:39:05.711045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:05.710986 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" podStartSLOduration=2.8081880249999998 podStartE2EDuration="6.710972133s" totalImagesPullingTime="3.902784108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:05.710834325 +0000 UTC m=+368.799170996" watchObservedRunningTime="2026-11-01 08:39:05.710972133 +0000 UTC m=+368.799308804" Nov 01 08:39:06.612082 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:06.612042 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" event={"ID":"308c91a4-a197-4420-8d55-7d7e9fe1e44d","Type":"ContainerStarted","Data":"8d506d11ae3022c5f5aa94a0d6ee24cba4c008718c266300ad03e8e7b664f3bb"} Nov 01 08:39:06.722167 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:06.722095 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" podStartSLOduration=2.452733668 podStartE2EDuration="4.722074046s" totalImagesPullingTime="2.269340378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:06.721696769 +0000 UTC m=+369.810033455" watchObservedRunningTime="2026-11-01 08:39:06.722074046 +0000 UTC m=+369.810410816" Nov 01 08:39:07.615890 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:07.615855 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:15.477967 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:39:15.477921 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:39:18.620230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:18.620201 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2rfzc" Nov 01 08:39:45.477075 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:39:45.477038 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:39:58.209172 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.209135 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh"] Nov 01 08:39:58.215010 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.214986 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.311998 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.311950 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh"] Nov 01 08:39:58.395209 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395169 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.395209 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395215 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.395529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395239 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.395529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395313 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5msk\" (UniqueName: \"kubernetes.io/projected/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-kube-api-access-g5msk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.395529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395380 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.395529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395473 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.395529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.395519 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.496807 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496681 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.496807 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496749 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.496807 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496773 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.496807 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496793 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5msk\" (UniqueName: \"kubernetes.io/projected/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-kube-api-access-g5msk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.497146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496826 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.497146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496875 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.497146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.496910 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.497637 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.497612 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.499370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.499342 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.499466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.499399 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.499732 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.499693 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.499819 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.499756 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.554819 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.554787 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.555005 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.554980 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5msk\" (UniqueName: \"kubernetes.io/projected/d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6-kube-api-access-g5msk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8wbh\" (UID: \"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:58.827404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:58.827306 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:59.265889 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.265852 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh"] Nov 01 08:39:59.273849 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:39:59.273814 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a861ef_ebbf_4b75_a0ab_58cd36f8b8a6.slice/crio-82baf3bee70888826320a917d6a69054e3143491d009f6818c249dea85895802 WatchSource:0}: Error finding container 82baf3bee70888826320a917d6a69054e3143491d009f6818c249dea85895802: Status 404 returned error can't find the container with id 82baf3bee70888826320a917d6a69054e3143491d009f6818c249dea85895802 Nov 01 08:39:59.275932 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.275894 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:39:59.276026 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.275979 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:39:59.837316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.837275 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" event={"ID":"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6","Type":"ContainerStarted","Data":"e8cdbe6c63e47aeb3242e1ff54c01b2772c35ebedecd8a377cd0d3bc558b0ee5"} Nov 01 08:39:59.837523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.837327 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" event={"ID":"d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6","Type":"ContainerStarted","Data":"82baf3bee70888826320a917d6a69054e3143491d009f6818c249dea85895802"} Nov 01 08:39:59.837523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.837387 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:39:59.982514 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:39:59.982442 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" podStartSLOduration=1.982425077 podStartE2EDuration="1.982425077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:39:59.981879491 +0000 UTC m=+423.070216164" watchObservedRunningTime="2026-11-01 08:39:59.982425077 +0000 UTC m=+423.070761753" Nov 01 08:40:00.843816 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:00.843785 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8wbh" Nov 01 08:40:01.059643 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.059609 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9"] Nov 01 08:40:01.059944 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.059902 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" containerName="discovery" containerID="cri-o://963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d" gracePeriod=30 Nov 01 08:40:01.311648 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.311624 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425634 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425695 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425755 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425801 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425858 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425902 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.426585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.425934 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw\") pod \"2c43382d-ec74-4862-b34e-76efe2587f37\" (UID: \"2c43382d-ec74-4862-b34e-76efe2587f37\") " Nov 01 08:40:01.432479 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.432119 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:40:01.433615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.433252 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:40:01.435817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.434376 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:40:01.435817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.434389 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "kube-api-access-p65cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:40:01.437679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.437635 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:40:01.437679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.437648 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:40:01.437939 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.437771 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token" pod "2c43382d-ec74-4862-b34e-76efe2587f37" (UID: "2c43382d-ec74-4862-b34e-76efe2587f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:40:01.527600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527560 5710 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-cacerts\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.527600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527594 5710 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-kubeconfig\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.527600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527607 5710 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-dns-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.527901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527622 5710 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2c43382d-ec74-4862-b34e-76efe2587f37-istio-csr-ca-configmap\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.527901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527637 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.527901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527649 5710 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c43382d-ec74-4862-b34e-76efe2587f37-local-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.527901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.527660 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65cw\" (UniqueName: \"kubernetes.io/projected/2c43382d-ec74-4862-b34e-76efe2587f37-kube-api-access-p65cw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:01.847373 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.847341 5710 generic.go:356] "Generic (PLEG): container finished" podID="2c43382d-ec74-4862-b34e-76efe2587f37" containerID="963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d" exitCode=0 Nov 01 08:40:01.847825 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.847417 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" Nov 01 08:40:01.847825 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.847436 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" event={"ID":"2c43382d-ec74-4862-b34e-76efe2587f37","Type":"ContainerDied","Data":"963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d"} Nov 01 08:40:01.847825 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.847521 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9" event={"ID":"2c43382d-ec74-4862-b34e-76efe2587f37","Type":"ContainerDied","Data":"f94e0d37718acdff7604f0e1758146845c5645f288754e31f0de2bbec7b14695"} Nov 01 08:40:01.847825 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.847549 5710 scope.go:122] "RemoveContainer" containerID="963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d" Nov 01 08:40:01.856652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.856630 5710 scope.go:122] "RemoveContainer" containerID="963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d" Nov 01 08:40:01.856951 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:40:01.856926 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d\": container with ID starting with 963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d not found: ID does not exist" containerID="963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d" Nov 01 08:40:01.857021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:01.856963 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d"} err="failed to get container status \"963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d\": rpc error: code = NotFound desc = could not find container \"963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d\": container with ID starting with 963e1b01e919ab3ec907ffb696426326f215bb51933fa4ef10059c92d6a2fa5d not found: ID does not exist" Nov 01 08:40:02.039970 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:02.039932 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9"] Nov 01 08:40:02.070355 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:02.070316 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-swgd9"] Nov 01 08:40:03.056917 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.056880 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" path="/var/lib/kubelet/pods/2c43382d-ec74-4862-b34e-76efe2587f37/volumes" Nov 01 08:40:03.816464 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.816431 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5w7w2"] Nov 01 08:40:03.816800 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.816786 5710 cpu_manager.go:414] "containerMap: removing container" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" containerName="discovery" Nov 01 08:40:03.816855 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.816801 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" containerName="discovery" Nov 01 08:40:03.816889 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.816875 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c43382d-ec74-4862-b34e-76efe2587f37" containerName="discovery" Nov 01 08:40:03.821417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.821394 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:03.872936 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.872903 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 08:40:03.874246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.874210 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5w7w2"] Nov 01 08:40:03.919347 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.919312 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 08:40:03.919493 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.919356 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 08:40:03.949933 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.949882 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85247c14-ee89-4497-9d63-7518733b0a98-cert\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:03.950129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.949995 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgslm\" (UniqueName: \"kubernetes.io/projected/85247c14-ee89-4497-9d63-7518733b0a98-kube-api-access-pgslm\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:03.950129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.950097 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85247c14-ee89-4497-9d63-7518733b0a98-metrics-certs\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:03.952229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.952191 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 08:40:03.952229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:03.952191 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wrsqt\"" Nov 01 08:40:04.051486 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.051450 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85247c14-ee89-4497-9d63-7518733b0a98-cert\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.051698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.051499 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgslm\" (UniqueName: \"kubernetes.io/projected/85247c14-ee89-4497-9d63-7518733b0a98-kube-api-access-pgslm\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.051698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.051580 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85247c14-ee89-4497-9d63-7518733b0a98-metrics-certs\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.054231 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.054205 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85247c14-ee89-4497-9d63-7518733b0a98-metrics-certs\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.054374 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.054353 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85247c14-ee89-4497-9d63-7518733b0a98-cert\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.109150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.109111 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgslm\" (UniqueName: \"kubernetes.io/projected/85247c14-ee89-4497-9d63-7518733b0a98-kube-api-access-pgslm\") pod \"odh-model-controller-769875db56-5w7w2\" (UID: \"85247c14-ee89-4497-9d63-7518733b0a98\") " pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.134983 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.134952 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:04.504407 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.504320 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5w7w2"] Nov 01 08:40:04.512748 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:40:04.512694 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85247c14_ee89_4497_9d63_7518733b0a98.slice/crio-cbc27330f941b6fc72a1b474912a1716e02d8ee29b31620665ce68d07464d8a1 WatchSource:0}: Error finding container cbc27330f941b6fc72a1b474912a1716e02d8ee29b31620665ce68d07464d8a1: Status 404 returned error can't find the container with id cbc27330f941b6fc72a1b474912a1716e02d8ee29b31620665ce68d07464d8a1 Nov 01 08:40:04.863804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:04.863770 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5w7w2" event={"ID":"85247c14-ee89-4497-9d63-7518733b0a98","Type":"ContainerStarted","Data":"cbc27330f941b6fc72a1b474912a1716e02d8ee29b31620665ce68d07464d8a1"} Nov 01 08:40:07.883527 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:07.883486 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5w7w2" event={"ID":"85247c14-ee89-4497-9d63-7518733b0a98","Type":"ContainerStarted","Data":"5094c102f29a66442c83315c3e724cc2eaec511a80d65e15b4c7b8d1bdccb3e5"} Nov 01 08:40:07.884002 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:07.883602 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:07.995100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:07.995040 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5w7w2" podStartSLOduration=1.7624123680000001 podStartE2EDuration="4.995024514s" totalImagesPullingTime="3.232612146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:07.994019211 +0000 UTC m=+431.082355885" watchObservedRunningTime="2026-11-01 08:40:07.995024514 +0000 UTC m=+431.083361237" Nov 01 08:40:15.476908 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:40:15.476865 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:40:18.890656 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:18.890621 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5w7w2" Nov 01 08:40:31.056240 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.056207 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67fc496566-fmc8w"] Nov 01 08:40:31.062666 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.062637 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.158922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.158869 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 08:40:31.158922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.158874 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 08:40:31.158922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.158874 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 08:40:31.159190 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.158878 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7zt8\"" Nov 01 08:40:31.159190 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.158932 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 08:40:31.160055 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.160035 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67fc496566-fmc8w"] Nov 01 08:40:31.195108 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.195066 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.195320 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.195128 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.195320 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.195264 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.195320 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.195296 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vszf\" (UniqueName: \"kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.296541 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.296493 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.296776 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.296558 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.296776 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.296586 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vszf\" (UniqueName: \"kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.296776 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.296648 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.297483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.297396 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.299381 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.299357 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.299524 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.299501 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.354894 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.354861 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vszf\" (UniqueName: \"kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf\") pod \"maas-api-67fc496566-fmc8w\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.376560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.376516 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:31.750296 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.750213 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67fc496566-fmc8w"] Nov 01 08:40:31.758770 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:40:31.758739 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd774acca_8c38_4fad_aa18_1025e60d1fcf.slice/crio-9056b3d54dcde077807a475ff039c367c40ee41f1688d65e25067a1947a8bb25 WatchSource:0}: Error finding container 9056b3d54dcde077807a475ff039c367c40ee41f1688d65e25067a1947a8bb25: Status 404 returned error can't find the container with id 9056b3d54dcde077807a475ff039c367c40ee41f1688d65e25067a1947a8bb25 Nov 01 08:40:31.995796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:31.995759 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" event={"ID":"d774acca-8c38-4fad-aa18-1025e60d1fcf","Type":"ContainerStarted","Data":"9056b3d54dcde077807a475ff039c367c40ee41f1688d65e25067a1947a8bb25"} Nov 01 08:40:36.226085 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:36.225991 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:40:39.031330 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:39.031289 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" event={"ID":"d774acca-8c38-4fad-aa18-1025e60d1fcf","Type":"ContainerStarted","Data":"90dc5c053f052d27c8d0b34d68d7c9c94865459b35c8bd7f81aa3fafdb9e7ae2"} Nov 01 08:40:39.031771 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:39.031426 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:39.142321 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:39.142267 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" podStartSLOduration=1.725471007 podStartE2EDuration="8.142252562s" totalImagesPullingTime="6.416781555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:39.141265721 +0000 UTC m=+462.229602395" watchObservedRunningTime="2026-11-01 08:40:39.142252562 +0000 UTC m=+462.230589233" Nov 01 08:40:45.041895 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:45.041857 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:45.476394 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:40:45.476357 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:40:46.228557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.228523 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f7bd94db-wfffb"] Nov 01 08:40:46.232079 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.232062 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:46.284788 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.284758 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7bd94db-wfffb"] Nov 01 08:40:46.340022 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.339976 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmrg\" (UniqueName: \"kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg\") pod \"authorino-5f7bd94db-wfffb\" (UID: \"0fba2f76-4172-409e-a3c4-48d0a42cdab0\") " pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:46.440976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.440934 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmrg\" (UniqueName: \"kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg\") pod \"authorino-5f7bd94db-wfffb\" (UID: \"0fba2f76-4172-409e-a3c4-48d0a42cdab0\") " pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:46.498472 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.498395 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmrg\" (UniqueName: \"kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg\") pod \"authorino-5f7bd94db-wfffb\" (UID: \"0fba2f76-4172-409e-a3c4-48d0a42cdab0\") " pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:46.543663 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.543616 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:46.697587 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.697545 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7bd94db-wfffb"] Nov 01 08:40:46.916997 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:46.916957 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7bd94db-wfffb"] Nov 01 08:40:46.921017 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:40:46.920985 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fba2f76_4172_409e_a3c4_48d0a42cdab0.slice/crio-69adeed331cb9b67e2d57e90200d965cdd6328a88103185bc03a77452478a19e WatchSource:0}: Error finding container 69adeed331cb9b67e2d57e90200d965cdd6328a88103185bc03a77452478a19e: Status 404 returned error can't find the container with id 69adeed331cb9b67e2d57e90200d965cdd6328a88103185bc03a77452478a19e Nov 01 08:40:47.067124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:47.067089 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7bd94db-wfffb" event={"ID":"0fba2f76-4172-409e-a3c4-48d0a42cdab0","Type":"ContainerStarted","Data":"69adeed331cb9b67e2d57e90200d965cdd6328a88103185bc03a77452478a19e"} Nov 01 08:40:49.076873 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.076782 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7bd94db-wfffb" event={"ID":"0fba2f76-4172-409e-a3c4-48d0a42cdab0","Type":"ContainerStarted","Data":"b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0"} Nov 01 08:40:49.076873 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.076832 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f7bd94db-wfffb" podUID="0fba2f76-4172-409e-a3c4-48d0a42cdab0" containerName="authorino" containerID="cri-o://b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0" gracePeriod=30 Nov 01 08:40:49.184184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.184112 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f7bd94db-wfffb" podStartSLOduration=1.313341181 podStartE2EDuration="3.18409187s" totalImagesPullingTime="1.870750689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:40:49.183375884 +0000 UTC m=+472.271712557" watchObservedRunningTime="2026-11-01 08:40:49.18409187 +0000 UTC m=+472.272428545" Nov 01 08:40:49.330379 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.330313 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:49.364435 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.364399 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg\" (UniqueName: \"kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg\") pod \"0fba2f76-4172-409e-a3c4-48d0a42cdab0\" (UID: \"0fba2f76-4172-409e-a3c4-48d0a42cdab0\") " Nov 01 08:40:49.366814 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.366784 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg" pod "0fba2f76-4172-409e-a3c4-48d0a42cdab0" (UID: "0fba2f76-4172-409e-a3c4-48d0a42cdab0"). InnerVolumeSpecName "kube-api-access-8zmrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:40:49.465926 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.465878 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmrg\" (UniqueName: \"kubernetes.io/projected/0fba2f76-4172-409e-a3c4-48d0a42cdab0-kube-api-access-8zmrg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:49.875499 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.875465 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kmlrt"] Nov 01 08:40:49.875769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:49.875647 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" podUID="81fe0ef3-04a5-45f1-a67c-204ff2bd3293" containerName="authorino" containerID="cri-o://ae543a9edab820052109380875fbdb1d85a3475b91e3702b5558f52f460e0637" gracePeriod=30 Nov 01 08:40:50.083693 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.083651 5710 generic.go:356] "Generic (PLEG): container finished" podID="81fe0ef3-04a5-45f1-a67c-204ff2bd3293" containerID="ae543a9edab820052109380875fbdb1d85a3475b91e3702b5558f52f460e0637" exitCode=0 Nov 01 08:40:50.084104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.083748 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" event={"ID":"81fe0ef3-04a5-45f1-a67c-204ff2bd3293","Type":"ContainerDied","Data":"ae543a9edab820052109380875fbdb1d85a3475b91e3702b5558f52f460e0637"} Nov 01 08:40:50.085105 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.085078 5710 generic.go:356] "Generic (PLEG): container finished" podID="0fba2f76-4172-409e-a3c4-48d0a42cdab0" containerID="b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0" exitCode=0 Nov 01 08:40:50.085246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.085122 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7bd94db-wfffb" Nov 01 08:40:50.085246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.085157 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7bd94db-wfffb" event={"ID":"0fba2f76-4172-409e-a3c4-48d0a42cdab0","Type":"ContainerDied","Data":"b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0"} Nov 01 08:40:50.085246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.085201 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7bd94db-wfffb" event={"ID":"0fba2f76-4172-409e-a3c4-48d0a42cdab0","Type":"ContainerDied","Data":"69adeed331cb9b67e2d57e90200d965cdd6328a88103185bc03a77452478a19e"} Nov 01 08:40:50.085246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.085223 5710 scope.go:122] "RemoveContainer" containerID="b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0" Nov 01 08:40:50.094310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.094281 5710 scope.go:122] "RemoveContainer" containerID="b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0" Nov 01 08:40:50.094782 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:40:50.094752 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0\": container with ID starting with b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0 not found: ID does not exist" containerID="b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0" Nov 01 08:40:50.094933 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.094791 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0"} err="failed to get container status \"b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0\": rpc error: code = NotFound desc = could not find container \"b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0\": container with ID starting with b1a3f8b240b2cd096b74490e24cafc0249807d39728501765a8d899d915d0af0 not found: ID does not exist" Nov 01 08:40:50.124366 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.124343 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:40:50.171383 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.171281 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7\" (UniqueName: \"kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7\") pod \"81fe0ef3-04a5-45f1-a67c-204ff2bd3293\" (UID: \"81fe0ef3-04a5-45f1-a67c-204ff2bd3293\") " Nov 01 08:40:50.173553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.173521 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7" pod "81fe0ef3-04a5-45f1-a67c-204ff2bd3293" (UID: "81fe0ef3-04a5-45f1-a67c-204ff2bd3293"). InnerVolumeSpecName "kube-api-access-lszx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:40:50.236852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.236816 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7bd94db-wfffb"] Nov 01 08:40:50.243214 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.243146 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f7bd94db-wfffb"] Nov 01 08:40:50.271953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:50.271910 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lszx7\" (UniqueName: \"kubernetes.io/projected/81fe0ef3-04a5-45f1-a67c-204ff2bd3293-kube-api-access-lszx7\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:51.056227 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:51.056191 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fba2f76-4172-409e-a3c4-48d0a42cdab0" path="/var/lib/kubelet/pods/0fba2f76-4172-409e-a3c4-48d0a42cdab0/volumes" Nov 01 08:40:51.090418 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:51.090384 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" Nov 01 08:40:51.090418 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:51.090401 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kmlrt" event={"ID":"81fe0ef3-04a5-45f1-a67c-204ff2bd3293","Type":"ContainerDied","Data":"5f77ee1e81d2d9bb871a753b008b73e9e3de5d999a0e7fe9668f8ef9a1f93456"} Nov 01 08:40:51.090994 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:51.090452 5710 scope.go:122] "RemoveContainer" containerID="ae543a9edab820052109380875fbdb1d85a3475b91e3702b5558f52f460e0637" Nov 01 08:40:51.241765 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:51.241676 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kmlrt"] Nov 01 08:40:51.247351 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:51.247316 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kmlrt"] Nov 01 08:40:53.056544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:53.056510 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fe0ef3-04a5-45f1-a67c-204ff2bd3293" path="/var/lib/kubelet/pods/81fe0ef3-04a5-45f1-a67c-204ff2bd3293/volumes" Nov 01 08:40:57.912108 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:57.912074 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67fc496566-fmc8w"] Nov 01 08:40:57.912567 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:57.912329 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" podUID="d774acca-8c38-4fad-aa18-1025e60d1fcf" containerName="maas-api" containerID="cri-o://90dc5c053f052d27c8d0b34d68d7c9c94865459b35c8bd7f81aa3fafdb9e7ae2" gracePeriod=30 Nov 01 08:40:58.125683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.125651 5710 generic.go:356] "Generic (PLEG): container finished" podID="d774acca-8c38-4fad-aa18-1025e60d1fcf" containerID="90dc5c053f052d27c8d0b34d68d7c9c94865459b35c8bd7f81aa3fafdb9e7ae2" exitCode=0 Nov 01 08:40:58.125882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.125736 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" event={"ID":"d774acca-8c38-4fad-aa18-1025e60d1fcf","Type":"ContainerDied","Data":"90dc5c053f052d27c8d0b34d68d7c9c94865459b35c8bd7f81aa3fafdb9e7ae2"} Nov 01 08:40:58.159568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.159541 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:58.243301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.243208 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf\" (UniqueName: \"kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf\") pod \"d774acca-8c38-4fad-aa18-1025e60d1fcf\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " Nov 01 08:40:58.243301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.243275 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca\") pod \"d774acca-8c38-4fad-aa18-1025e60d1fcf\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " Nov 01 08:40:58.243301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.243301 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs\") pod \"d774acca-8c38-4fad-aa18-1025e60d1fcf\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " Nov 01 08:40:58.243586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.243352 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls\") pod \"d774acca-8c38-4fad-aa18-1025e60d1fcf\" (UID: \"d774acca-8c38-4fad-aa18-1025e60d1fcf\") " Nov 01 08:40:58.243771 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.243745 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca" pod "d774acca-8c38-4fad-aa18-1025e60d1fcf" (UID: "d774acca-8c38-4fad-aa18-1025e60d1fcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:40:58.245872 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.245836 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs" pod "d774acca-8c38-4fad-aa18-1025e60d1fcf" (UID: "d774acca-8c38-4fad-aa18-1025e60d1fcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:40:58.245991 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.245903 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls" pod "d774acca-8c38-4fad-aa18-1025e60d1fcf" (UID: "d774acca-8c38-4fad-aa18-1025e60d1fcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:40:58.245991 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.245979 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf" pod "d774acca-8c38-4fad-aa18-1025e60d1fcf" (UID: "d774acca-8c38-4fad-aa18-1025e60d1fcf"). InnerVolumeSpecName "kube-api-access-9vszf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:40:58.343974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.343940 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vszf\" (UniqueName: \"kubernetes.io/projected/d774acca-8c38-4fad-aa18-1025e60d1fcf-kube-api-access-9vszf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:58.343974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.343969 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d774acca-8c38-4fad-aa18-1025e60d1fcf-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:58.343974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.343979 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:58.344232 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:58.343987 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d774acca-8c38-4fad-aa18-1025e60d1fcf-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:40:59.132305 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:59.132280 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" Nov 01 08:40:59.132769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:59.132277 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67fc496566-fmc8w" event={"ID":"d774acca-8c38-4fad-aa18-1025e60d1fcf","Type":"ContainerDied","Data":"9056b3d54dcde077807a475ff039c367c40ee41f1688d65e25067a1947a8bb25"} Nov 01 08:40:59.132769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:59.132418 5710 scope.go:122] "RemoveContainer" containerID="90dc5c053f052d27c8d0b34d68d7c9c94865459b35c8bd7f81aa3fafdb9e7ae2" Nov 01 08:40:59.285271 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:59.285228 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67fc496566-fmc8w"] Nov 01 08:40:59.290626 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:40:59.290589 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67fc496566-fmc8w"] Nov 01 08:41:01.056170 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:01.056128 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d774acca-8c38-4fad-aa18-1025e60d1fcf" path="/var/lib/kubelet/pods/d774acca-8c38-4fad-aa18-1025e60d1fcf/volumes" Nov 01 08:41:15.477820 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:41:15.477776 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:41:45.143771 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.143702 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fpl8d"] Nov 01 08:41:45.144287 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144264 5710 cpu_manager.go:414] "containerMap: removing container" podUID="81fe0ef3-04a5-45f1-a67c-204ff2bd3293" containerName="authorino" Nov 01 08:41:45.144287 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144287 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fe0ef3-04a5-45f1-a67c-204ff2bd3293" containerName="authorino" Nov 01 08:41:45.144443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144300 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d774acca-8c38-4fad-aa18-1025e60d1fcf" containerName="maas-api" Nov 01 08:41:45.144443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144311 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d774acca-8c38-4fad-aa18-1025e60d1fcf" containerName="maas-api" Nov 01 08:41:45.144443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144354 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0fba2f76-4172-409e-a3c4-48d0a42cdab0" containerName="authorino" Nov 01 08:41:45.144443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144362 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fba2f76-4172-409e-a3c4-48d0a42cdab0" containerName="authorino" Nov 01 08:41:45.144567 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144459 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fe0ef3-04a5-45f1-a67c-204ff2bd3293" containerName="authorino" Nov 01 08:41:45.144567 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144469 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d774acca-8c38-4fad-aa18-1025e60d1fcf" containerName="maas-api" Nov 01 08:41:45.144567 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.144479 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fba2f76-4172-409e-a3c4-48d0a42cdab0" containerName="authorino" Nov 01 08:41:45.148114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.148087 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.248530 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.248475 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 08:41:45.248798 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.248589 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 08:41:45.248798 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.248666 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-c2rlq\"" Nov 01 08:41:45.249653 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.249613 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fpl8d"] Nov 01 08:41:45.272867 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.272832 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-fpl8d\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.273061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.272967 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdcq\" (UniqueName: \"kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq\") pod \"ai-gateway-controller-7d85675f98-fpl8d\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.373837 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.373801 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-fpl8d\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.374070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.373866 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdcq\" (UniqueName: \"kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq\") pod \"ai-gateway-controller-7d85675f98-fpl8d\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.376710 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.376677 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-fpl8d\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.432367 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.432286 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdcq\" (UniqueName: \"kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq\") pod \"ai-gateway-controller-7d85675f98-fpl8d\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.463789 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.463753 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:45.478210 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:41:45.478113 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:41:45.837194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:45.837110 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fpl8d"] Nov 01 08:41:45.845224 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:41:45.845196 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd315163f_7f9d_435e_b704_8b6befc13180.slice/crio-3ab359f74566fd37961d070ce116e943957266e1db226c3e4b2d21c722f846ef WatchSource:0}: Error finding container 3ab359f74566fd37961d070ce116e943957266e1db226c3e4b2d21c722f846ef: Status 404 returned error can't find the container with id 3ab359f74566fd37961d070ce116e943957266e1db226c3e4b2d21c722f846ef Nov 01 08:41:46.336709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:46.336673 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" event={"ID":"d315163f-7f9d-435e-b704-8b6befc13180","Type":"ContainerStarted","Data":"3ab359f74566fd37961d070ce116e943957266e1db226c3e4b2d21c722f846ef"} Nov 01 08:41:48.345996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:48.345961 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" event={"ID":"d315163f-7f9d-435e-b704-8b6befc13180","Type":"ContainerStarted","Data":"90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae"} Nov 01 08:41:48.346421 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:48.346113 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:48.455363 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:48.455306 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" podStartSLOduration=1.345640175 podStartE2EDuration="3.455284284s" totalImagesPullingTime="2.109644109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:41:48.454542914 +0000 UTC m=+531.542879579" watchObservedRunningTime="2026-11-01 08:41:48.455284284 +0000 UTC m=+531.543620957" Nov 01 08:41:52.146586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.146546 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6"] Nov 01 08:41:52.150141 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.150121 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.198914 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.198877 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 08:41:52.238306 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.238264 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7hrd6\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.238495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.238403 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jsk\" (UniqueName: \"kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk\") pod \"payload-pre-processing-7b98bb678c-7hrd6\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.245243 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.245219 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4h9j5\"" Nov 01 08:41:52.246258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.246234 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6"] Nov 01 08:41:52.299739 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.299695 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sccmk"] Nov 01 08:41:52.303299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.303275 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.338925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.338897 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7hrd6\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.339098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.338970 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jsk\" (UniqueName: \"kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk\") pod \"payload-pre-processing-7b98bb678c-7hrd6\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.339584 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.339566 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7hrd6\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.357255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.357222 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sccmk"] Nov 01 08:41:52.396887 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.396808 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jsk\" (UniqueName: \"kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk\") pod \"payload-pre-processing-7b98bb678c-7hrd6\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.440543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.440233 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-sccmk\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.440543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.440412 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzqb9\" (UniqueName: \"kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9\") pod \"payload-processing-55fbbbf9dc-sccmk\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.463762 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.463695 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:52.541412 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.541377 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzqb9\" (UniqueName: \"kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9\") pod \"payload-processing-55fbbbf9dc-sccmk\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.541579 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.541439 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-sccmk\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.542266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.542238 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-sccmk\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.598964 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.598933 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzqb9\" (UniqueName: \"kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9\") pod \"payload-processing-55fbbbf9dc-sccmk\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.616841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.616810 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:52.839202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.839166 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6"] Nov 01 08:41:52.847215 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:41:52.847186 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ef7fd8_4cd8_4e62_b4f9_45a690ea7b1c.slice/crio-6d5e75dc60d609c6eb303ee35d77c700e65de169d733069e45132ebe16e93ac7 WatchSource:0}: Error finding container 6d5e75dc60d609c6eb303ee35d77c700e65de169d733069e45132ebe16e93ac7: Status 404 returned error can't find the container with id 6d5e75dc60d609c6eb303ee35d77c700e65de169d733069e45132ebe16e93ac7 Nov 01 08:41:52.993093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:52.993007 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sccmk"] Nov 01 08:41:53.001141 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:41:53.001107 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3dd39c_5a0b_4ad0_a84d_c826fa2d0378.slice/crio-96087825c53ad653c97e71cd2ef91b0076a19c228abc883557aee0f8d027aa12 WatchSource:0}: Error finding container 96087825c53ad653c97e71cd2ef91b0076a19c228abc883557aee0f8d027aa12: Status 404 returned error can't find the container with id 96087825c53ad653c97e71cd2ef91b0076a19c228abc883557aee0f8d027aa12 Nov 01 08:41:53.370054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:53.370020 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" event={"ID":"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c","Type":"ContainerStarted","Data":"6d5e75dc60d609c6eb303ee35d77c700e65de169d733069e45132ebe16e93ac7"} Nov 01 08:41:53.371235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:53.371211 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" event={"ID":"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378","Type":"ContainerStarted","Data":"96087825c53ad653c97e71cd2ef91b0076a19c228abc883557aee0f8d027aa12"} Nov 01 08:41:56.389235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:56.389197 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" event={"ID":"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378","Type":"ContainerStarted","Data":"a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824"} Nov 01 08:41:56.389713 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:56.389353 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:41:56.390803 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:56.390786 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" event={"ID":"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c","Type":"ContainerStarted","Data":"747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe"} Nov 01 08:41:56.390920 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:56.390907 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:41:56.498266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:56.498214 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" podStartSLOduration=1.582004463 podStartE2EDuration="4.498195168s" totalImagesPullingTime="2.916190705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:41:56.497434108 +0000 UTC m=+539.585770792" watchObservedRunningTime="2026-11-01 08:41:56.498195168 +0000 UTC m=+539.586531843" Nov 01 08:41:56.606513 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:56.606461 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" podStartSLOduration=1.503349453 podStartE2EDuration="4.606445765s" totalImagesPullingTime="3.103096312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:41:56.605433451 +0000 UTC m=+539.693770123" watchObservedRunningTime="2026-11-01 08:41:56.606445765 +0000 UTC m=+539.694782437" Nov 01 08:41:59.352811 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.352779 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:41:59.707565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.707488 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7657794547-t5tw5"] Nov 01 08:41:59.712508 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.712483 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:41:59.768080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.768045 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7657794547-t5tw5"] Nov 01 08:41:59.811596 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.811564 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7ffc9a-2189-49be-adcb-2ff757eb57d5-metrics-certs\") pod \"ai-gateway-controller-7657794547-t5tw5\" (UID: \"ab7ffc9a-2189-49be-adcb-2ff757eb57d5\") " pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:41:59.811596 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.811601 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ck7v\" (UniqueName: \"kubernetes.io/projected/ab7ffc9a-2189-49be-adcb-2ff757eb57d5-kube-api-access-5ck7v\") pod \"ai-gateway-controller-7657794547-t5tw5\" (UID: \"ab7ffc9a-2189-49be-adcb-2ff757eb57d5\") " pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:41:59.913191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.913149 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7ffc9a-2189-49be-adcb-2ff757eb57d5-metrics-certs\") pod \"ai-gateway-controller-7657794547-t5tw5\" (UID: \"ab7ffc9a-2189-49be-adcb-2ff757eb57d5\") " pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:41:59.913191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.913193 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ck7v\" (UniqueName: \"kubernetes.io/projected/ab7ffc9a-2189-49be-adcb-2ff757eb57d5-kube-api-access-5ck7v\") pod \"ai-gateway-controller-7657794547-t5tw5\" (UID: \"ab7ffc9a-2189-49be-adcb-2ff757eb57d5\") " pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:41:59.915932 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.915906 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7ffc9a-2189-49be-adcb-2ff757eb57d5-metrics-certs\") pod \"ai-gateway-controller-7657794547-t5tw5\" (UID: \"ab7ffc9a-2189-49be-adcb-2ff757eb57d5\") " pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:41:59.970988 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:41:59.970896 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ck7v\" (UniqueName: \"kubernetes.io/projected/ab7ffc9a-2189-49be-adcb-2ff757eb57d5-kube-api-access-5ck7v\") pod \"ai-gateway-controller-7657794547-t5tw5\" (UID: \"ab7ffc9a-2189-49be-adcb-2ff757eb57d5\") " pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:42:00.025550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:00.025508 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:42:00.394601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:00.394564 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7657794547-t5tw5"] Nov 01 08:42:00.403887 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:00.403854 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7ffc9a_2189_49be_adcb_2ff757eb57d5.slice/crio-5810bb06b6e39eef0934a8516ff96f905bf4da49e3fd850f02c0369ee33f323d WatchSource:0}: Error finding container 5810bb06b6e39eef0934a8516ff96f905bf4da49e3fd850f02c0369ee33f323d: Status 404 returned error can't find the container with id 5810bb06b6e39eef0934a8516ff96f905bf4da49e3fd850f02c0369ee33f323d Nov 01 08:42:00.409909 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:00.409860 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" event={"ID":"ab7ffc9a-2189-49be-adcb-2ff757eb57d5","Type":"ContainerStarted","Data":"5810bb06b6e39eef0934a8516ff96f905bf4da49e3fd850f02c0369ee33f323d"} Nov 01 08:42:01.416257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:01.416218 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" event={"ID":"ab7ffc9a-2189-49be-adcb-2ff757eb57d5","Type":"ContainerStarted","Data":"42cb2f3724656cac3d8c81e908db718ff53bfd55bd27b4d212b3b2be16863cb7"} Nov 01 08:42:01.416753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:01.416345 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:42:01.525668 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:01.525596 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" podStartSLOduration=1.650361763 podStartE2EDuration="2.525576873s" totalImagesPullingTime="875.21511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:01.525441821 +0000 UTC m=+544.613778497" watchObservedRunningTime="2026-11-01 08:42:01.525576873 +0000 UTC m=+544.613913544" Nov 01 08:42:07.398645 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:07.398616 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 08:42:07.399080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:07.398684 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 08:42:12.423809 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.423777 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7657794547-t5tw5" Nov 01 08:42:12.582476 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.582441 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fpl8d"] Nov 01 08:42:12.582999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.582952 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" podUID="d315163f-7f9d-435e-b704-8b6befc13180" containerName="manager" containerID="cri-o://90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae" gracePeriod=30 Nov 01 08:42:12.833233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.833207 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:42:12.930077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.930024 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs\") pod \"d315163f-7f9d-435e-b704-8b6befc13180\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " Nov 01 08:42:12.930288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.930142 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq\" (UniqueName: \"kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq\") pod \"d315163f-7f9d-435e-b704-8b6befc13180\" (UID: \"d315163f-7f9d-435e-b704-8b6befc13180\") " Nov 01 08:42:12.932519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.932489 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq" pod "d315163f-7f9d-435e-b704-8b6befc13180" (UID: "d315163f-7f9d-435e-b704-8b6befc13180"). InnerVolumeSpecName "kube-api-access-lbdcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:42:12.932606 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:12.932497 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs" pod "d315163f-7f9d-435e-b704-8b6befc13180" (UID: "d315163f-7f9d-435e-b704-8b6befc13180"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:42:13.031087 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.030994 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d315163f-7f9d-435e-b704-8b6befc13180-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:42:13.031087 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.031030 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdcq\" (UniqueName: \"kubernetes.io/projected/d315163f-7f9d-435e-b704-8b6befc13180-kube-api-access-lbdcq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:42:13.469891 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.469854 5710 generic.go:356] "Generic (PLEG): container finished" podID="d315163f-7f9d-435e-b704-8b6befc13180" containerID="90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae" exitCode=0 Nov 01 08:42:13.470434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.469921 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" Nov 01 08:42:13.470434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.469913 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" event={"ID":"d315163f-7f9d-435e-b704-8b6befc13180","Type":"ContainerDied","Data":"90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae"} Nov 01 08:42:13.470434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.470067 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-fpl8d" event={"ID":"d315163f-7f9d-435e-b704-8b6befc13180","Type":"ContainerDied","Data":"3ab359f74566fd37961d070ce116e943957266e1db226c3e4b2d21c722f846ef"} Nov 01 08:42:13.470434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.470103 5710 scope.go:122] "RemoveContainer" containerID="90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae" Nov 01 08:42:13.478902 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.478883 5710 scope.go:122] "RemoveContainer" containerID="90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae" Nov 01 08:42:13.479194 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:42:13.479173 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae\": container with ID starting with 90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae not found: ID does not exist" containerID="90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae" Nov 01 08:42:13.479242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.479205 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae"} err="failed to get container status \"90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae\": rpc error: code = NotFound desc = could not find container \"90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae\": container with ID starting with 90d4a175ff30e68500a99247e796d653dd8fee5f8506e37ce3c1569653725dae not found: ID does not exist" Nov 01 08:42:13.624171 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.624139 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fpl8d"] Nov 01 08:42:13.670280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:13.670246 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-fpl8d"] Nov 01 08:42:15.057173 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:15.057141 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d315163f-7f9d-435e-b704-8b6befc13180" path="/var/lib/kubelet/pods/d315163f-7f9d-435e-b704-8b6befc13180/volumes" Nov 01 08:42:15.478970 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:42:15.478926 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:42:22.444634 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.444597 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg"] Nov 01 08:42:22.445112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.445064 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d315163f-7f9d-435e-b704-8b6befc13180" containerName="manager" Nov 01 08:42:22.445112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.445076 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d315163f-7f9d-435e-b704-8b6befc13180" containerName="manager" Nov 01 08:42:22.445242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.445230 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d315163f-7f9d-435e-b704-8b6befc13180" containerName="manager" Nov 01 08:42:22.451771 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.451747 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.503072 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.502185 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 08:42:22.503072 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.502260 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 08:42:22.519171 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.519128 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.519342 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.519180 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.519342 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.519250 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxt7\" (UniqueName: \"kubernetes.io/projected/81739fa3-884f-402f-a010-2c5c8f2e4943-kube-api-access-plxt7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.519425 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.519383 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.519458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.519442 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81739fa3-884f-402f-a010-2c5c8f2e4943-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.519493 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.519469 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.548740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.548694 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 08:42:22.548946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.548928 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ffzjt\"" Nov 01 08:42:22.550130 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.550104 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg"] Nov 01 08:42:22.620674 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.620628 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.620903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.620697 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81739fa3-884f-402f-a010-2c5c8f2e4943-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.620903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.620752 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.620903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.620802 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.620903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.620828 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.620903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.620864 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxt7\" (UniqueName: \"kubernetes.io/projected/81739fa3-884f-402f-a010-2c5c8f2e4943-kube-api-access-plxt7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.621156 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.621089 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.621156 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.621141 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.621310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.621158 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.623188 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.623167 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81739fa3-884f-402f-a010-2c5c8f2e4943-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.623430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.623415 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81739fa3-884f-402f-a010-2c5c8f2e4943-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.678945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.678901 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxt7\" (UniqueName: \"kubernetes.io/projected/81739fa3-884f-402f-a010-2c5c8f2e4943-kube-api-access-plxt7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg\" (UID: \"81739fa3-884f-402f-a010-2c5c8f2e4943\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:22.765572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:22.765484 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:23.140447 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:23.140411 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg"] Nov 01 08:42:23.148773 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:23.148743 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81739fa3_884f_402f_a010_2c5c8f2e4943.slice/crio-2551e862feeb31897acdb6d2efa5414913d99f19b34c2704c1d28251f212e989 WatchSource:0}: Error finding container 2551e862feeb31897acdb6d2efa5414913d99f19b34c2704c1d28251f212e989: Status 404 returned error can't find the container with id 2551e862feeb31897acdb6d2efa5414913d99f19b34c2704c1d28251f212e989 Nov 01 08:42:23.521992 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:23.521901 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" event={"ID":"81739fa3-884f-402f-a010-2c5c8f2e4943","Type":"ContainerStarted","Data":"2551e862feeb31897acdb6d2efa5414913d99f19b34c2704c1d28251f212e989"} Nov 01 08:42:24.532135 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:24.532093 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:42:25.532134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:25.532026 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" event={"ID":"81739fa3-884f-402f-a010-2c5c8f2e4943","Type":"ContainerStarted","Data":"0ad0f73df458581d6bf08385bfe374ae640c5760b235341f3acc31b52845dd03"} Nov 01 08:42:25.646158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:25.646084 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" podStartSLOduration=1.518039837 podStartE2EDuration="3.646060446s" totalImagesPullingTime="2.128020609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:25.645073996 +0000 UTC m=+568.733410668" watchObservedRunningTime="2026-11-01 08:42:25.646060446 +0000 UTC m=+568.734397117" Nov 01 08:42:25.766273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:25.766225 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:31.209850 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.209811 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc"] Nov 01 08:42:31.214551 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.214513 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.302052 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.302000 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.302273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.302100 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.302273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.302135 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.302273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.302182 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw88j\" (UniqueName: \"kubernetes.io/projected/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-kube-api-access-tw88j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.302273 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.302256 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.302491 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.302295 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.309767 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.309735 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 08:42:31.311025 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.311000 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc"] Nov 01 08:42:31.403384 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403340 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.403384 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403388 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.403622 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403408 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.403622 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403427 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw88j\" (UniqueName: \"kubernetes.io/projected/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-kube-api-access-tw88j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.403622 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403485 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.403622 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403538 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.403919 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403883 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.404046 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403896 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.404046 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.403984 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.406010 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.405985 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.406200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.406183 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.463891 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.463821 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw88j\" (UniqueName: \"kubernetes.io/projected/8a5af8c7-0109-4d77-ab61-837d1b8f79e9-kube-api-access-tw88j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ct5sc\" (UID: \"8a5af8c7-0109-4d77-ab61-837d1b8f79e9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.528368 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.528324 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:31.907562 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:31.907523 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc"] Nov 01 08:42:31.916023 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:31.915989 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5af8c7_0109_4d77_ab61_837d1b8f79e9.slice/crio-888896c43f0fdba1ffa07bc82047733ea6bdc8838e1d138d445df405814e8e2c WatchSource:0}: Error finding container 888896c43f0fdba1ffa07bc82047733ea6bdc8838e1d138d445df405814e8e2c: Status 404 returned error can't find the container with id 888896c43f0fdba1ffa07bc82047733ea6bdc8838e1d138d445df405814e8e2c Nov 01 08:42:32.567381 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:32.567352 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" event={"ID":"8a5af8c7-0109-4d77-ab61-837d1b8f79e9","Type":"ContainerStarted","Data":"888896c43f0fdba1ffa07bc82047733ea6bdc8838e1d138d445df405814e8e2c"} Nov 01 08:42:32.765650 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:32.765611 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:32.778741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:32.778688 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:33.573798 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:33.573762 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" event={"ID":"8a5af8c7-0109-4d77-ab61-837d1b8f79e9","Type":"ContainerStarted","Data":"6b35f844437376e74faa9b085a3eb451191e1db70367d0c689c8d0fe2d6ab673"} Nov 01 08:42:33.586072 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:33.586035 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg" Nov 01 08:42:33.684152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:33.684091 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" podStartSLOduration=2.04011202 podStartE2EDuration="2.684072992s" totalImagesPullingTime="643.960972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:33.683626018 +0000 UTC m=+576.771962692" watchObservedRunningTime="2026-11-01 08:42:33.684072992 +0000 UTC m=+576.772409663" Nov 01 08:42:34.261736 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.261686 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp"] Nov 01 08:42:34.265457 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.265424 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.314414 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.314362 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 08:42:34.329248 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.329209 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.329248 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.329253 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2579b5-5533-45d8-a0b6-c4272b055f20-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.329510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.329346 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.329510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.329387 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqs9k\" (UniqueName: \"kubernetes.io/projected/0c2579b5-5533-45d8-a0b6-c4272b055f20-kube-api-access-xqs9k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.329510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.329475 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.329677 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.329576 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.361882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.361829 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp"] Nov 01 08:42:34.430183 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430139 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430183 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430186 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2579b5-5533-45d8-a0b6-c4272b055f20-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430220 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430238 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqs9k\" (UniqueName: \"kubernetes.io/projected/0c2579b5-5533-45d8-a0b6-c4272b055f20-kube-api-access-xqs9k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430354 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430426 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430604 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430680 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430665 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.430812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.430790 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.432867 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.432845 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2579b5-5533-45d8-a0b6-c4272b055f20-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.433114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.433095 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2579b5-5533-45d8-a0b6-c4272b055f20-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.488328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.488293 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqs9k\" (UniqueName: \"kubernetes.io/projected/0c2579b5-5533-45d8-a0b6-c4272b055f20-kube-api-access-xqs9k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lnhsp\" (UID: \"0c2579b5-5533-45d8-a0b6-c4272b055f20\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.529471 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.529373 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:34.554606 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.554568 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:42:34.579437 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.579405 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:34.963562 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:34.963519 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp"] Nov 01 08:42:34.971265 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:34.971233 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2579b5_5533_45d8_a0b6_c4272b055f20.slice/crio-1d1664900e3e4cb986b8114dd065016f1a597b8fb18e039924a22bd4c816eb80 WatchSource:0}: Error finding container 1d1664900e3e4cb986b8114dd065016f1a597b8fb18e039924a22bd4c816eb80: Status 404 returned error can't find the container with id 1d1664900e3e4cb986b8114dd065016f1a597b8fb18e039924a22bd4c816eb80 Nov 01 08:42:35.583497 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:35.583462 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" event={"ID":"0c2579b5-5533-45d8-a0b6-c4272b055f20","Type":"ContainerStarted","Data":"1d1664900e3e4cb986b8114dd065016f1a597b8fb18e039924a22bd4c816eb80"} Nov 01 08:42:36.595305 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:36.595260 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" event={"ID":"0c2579b5-5533-45d8-a0b6-c4272b055f20","Type":"ContainerStarted","Data":"a5461bebc3d197b0e71480b434a3ef11963c867e604cc307753348a9ba62a7f2"} Nov 01 08:42:36.706989 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:36.706922 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" podStartSLOduration=2.104246966 podStartE2EDuration="2.706907213s" totalImagesPullingTime="602.660247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:36.705694403 +0000 UTC m=+579.794031077" watchObservedRunningTime="2026-11-01 08:42:36.706907213 +0000 UTC m=+579.795243894" Nov 01 08:42:37.580243 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:37.580210 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:40.589550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.589504 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj"] Nov 01 08:42:40.593345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.593319 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.681844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.681804 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 08:42:40.685107 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.685059 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj"] Nov 01 08:42:40.690580 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.690550 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.690755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.690609 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.690853 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.690823 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.690914 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.690871 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1298d16-27bd-4638-8969-39eef9ffd697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.690954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.690942 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.690993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.690967 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9tdn\" (UniqueName: \"kubernetes.io/projected/d1298d16-27bd-4638-8969-39eef9ffd697-kube-api-access-b9tdn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791434 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791503 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791521 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1298d16-27bd-4638-8969-39eef9ffd697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791550 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791569 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9tdn\" (UniqueName: \"kubernetes.io/projected/d1298d16-27bd-4638-8969-39eef9ffd697-kube-api-access-b9tdn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791894 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791869 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.791980 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791956 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.792026 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.791995 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.792026 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.792013 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.794238 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.794211 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1298d16-27bd-4638-8969-39eef9ffd697-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.794550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.794531 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1298d16-27bd-4638-8969-39eef9ffd697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.853191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.853146 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9tdn\" (UniqueName: \"kubernetes.io/projected/d1298d16-27bd-4638-8969-39eef9ffd697-kube-api-access-b9tdn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj\" (UID: \"d1298d16-27bd-4638-8969-39eef9ffd697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:40.907030 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:40.906988 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:41.286043 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:41.285962 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj"] Nov 01 08:42:41.293948 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:41.293919 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1298d16_27bd_4638_8969_39eef9ffd697.slice/crio-e5c83cf3ceab6f2803d472a4603cac45083dee1fe35dd7c0a754defc6f9b278e WatchSource:0}: Error finding container e5c83cf3ceab6f2803d472a4603cac45083dee1fe35dd7c0a754defc6f9b278e: Status 404 returned error can't find the container with id e5c83cf3ceab6f2803d472a4603cac45083dee1fe35dd7c0a754defc6f9b278e Nov 01 08:42:41.529266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:41.529211 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:41.542191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:41.542097 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:41.620140 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:41.620106 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" event={"ID":"d1298d16-27bd-4638-8969-39eef9ffd697","Type":"ContainerStarted","Data":"e5c83cf3ceab6f2803d472a4603cac45083dee1fe35dd7c0a754defc6f9b278e"} Nov 01 08:42:41.631880 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:41.631836 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ct5sc" Nov 01 08:42:42.628663 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:42.628617 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" event={"ID":"d1298d16-27bd-4638-8969-39eef9ffd697","Type":"ContainerStarted","Data":"c28876735bef7724dc13ac05be540802a61a09dcb030d1a99c99f5e4851ccd9c"} Nov 01 08:42:42.739854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:42.739785 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" podStartSLOduration=2.170588791 podStartE2EDuration="2.739766388s" totalImagesPullingTime="569.177597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:42.7393743 +0000 UTC m=+585.827710973" watchObservedRunningTime="2026-11-01 08:42:42.739766388 +0000 UTC m=+585.828103058" Nov 01 08:42:42.907541 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:42.907448 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:44.580227 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:44.580191 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:44.592929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:44.592903 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:44.650525 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:44.650493 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" Nov 01 08:42:45.477051 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:42:45.477016 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:42:46.870351 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:46.870316 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:42:47.368677 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.368638 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8"] Nov 01 08:42:47.372568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.372536 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.425442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.425389 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8"] Nov 01 08:42:47.460543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.460504 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.460759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.460551 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-kube-api-access-2mnvk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.460759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.460666 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.460759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.460688 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.460759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.460711 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.460915 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.460775 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.469211 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.469187 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 08:42:47.562280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562228 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562293 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562330 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562354 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562472 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562754 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562505 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-kube-api-access-2mnvk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562757 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562774 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.562826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.562814 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.564888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.564860 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.565266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.565244 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.622424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.622279 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnvk\" (UniqueName: \"kubernetes.io/projected/814d41ce-7e8f-4d90-a1e4-8cde4feb47ce-kube-api-access-2mnvk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8\" (UID: \"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:47.686310 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:47.686266 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:48.063335 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:48.063247 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8"] Nov 01 08:42:48.071378 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:48.071345 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814d41ce_7e8f_4d90_a1e4_8cde4feb47ce.slice/crio-cc688de9d2288dfe03daaefb82005e4f4c636741492954783b73b99b5bc036b8 WatchSource:0}: Error finding container cc688de9d2288dfe03daaefb82005e4f4c636741492954783b73b99b5bc036b8: Status 404 returned error can't find the container with id cc688de9d2288dfe03daaefb82005e4f4c636741492954783b73b99b5bc036b8 Nov 01 08:42:48.665482 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:48.665448 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" event={"ID":"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce","Type":"ContainerStarted","Data":"cc688de9d2288dfe03daaefb82005e4f4c636741492954783b73b99b5bc036b8"} Nov 01 08:42:49.671576 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:49.671535 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" event={"ID":"814d41ce-7e8f-4d90-a1e4-8cde4feb47ce","Type":"ContainerStarted","Data":"4d490cc12bac18aba2a4c923639693fedc0ce8cdf5ed0ef86c971f1f67ce10c3"} Nov 01 08:42:49.687125 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:49.687095 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:49.782636 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:49.782581 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" podStartSLOduration=2.245645509 podStartE2EDuration="2.782566389s" totalImagesPullingTime="536.92088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:42:49.781215139 +0000 UTC m=+592.869551801" watchObservedRunningTime="2026-11-01 08:42:49.782566389 +0000 UTC m=+592.870903091" Nov 01 08:42:50.778095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.778062 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5"] Nov 01 08:42:50.781913 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.781883 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.868838 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.868798 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5"] Nov 01 08:42:50.893229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893193 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893448 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893242 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec42fe9-a657-4b42-bb54-7849913616b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893448 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893345 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893448 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893390 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893451 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctc6n\" (UniqueName: \"kubernetes.io/projected/eec42fe9-a657-4b42-bb54-7849913616b4-kube-api-access-ctc6n\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893529 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec42fe9-a657-4b42-bb54-7849913616b4-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893596 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893758 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893658 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.893758 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.893735 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec42fe9-a657-4b42-bb54-7849913616b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.908028 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.907988 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:50.914540 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.914513 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kpfx2\"" Nov 01 08:42:50.921040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.921018 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:50.994283 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994246 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994299 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994366 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec42fe9-a657-4b42-bb54-7849913616b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994429 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994454 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec42fe9-a657-4b42-bb54-7849913616b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994617 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994666 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994694 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctc6n\" (UniqueName: \"kubernetes.io/projected/eec42fe9-a657-4b42-bb54-7849913616b4-kube-api-access-ctc6n\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994702 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.994854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994784 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec42fe9-a657-4b42-bb54-7849913616b4-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.995007 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.994961 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.995328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.995308 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.995444 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.995354 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec42fe9-a657-4b42-bb54-7849913616b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.995515 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.995493 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.997016 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.996993 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec42fe9-a657-4b42-bb54-7849913616b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:50.997230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:50.997210 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec42fe9-a657-4b42-bb54-7849913616b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:51.052669 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:51.052583 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec42fe9-a657-4b42-bb54-7849913616b4-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:51.052829 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:51.052812 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctc6n\" (UniqueName: \"kubernetes.io/projected/eec42fe9-a657-4b42-bb54-7849913616b4-kube-api-access-ctc6n\") pod \"maas-default-gateway-openshift-default-746ff8f49f-gsql5\" (UID: \"eec42fe9-a657-4b42-bb54-7849913616b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:51.095854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:51.095800 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:42:51.523925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:51.523891 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5"] Nov 01 08:42:51.532580 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:42:51.532551 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec42fe9_a657_4b42_bb54_7849913616b4.slice/crio-27e6f776eca28eac3a949e115058559f022be8cda50b32a6906713f41eb46978 WatchSource:0}: Error finding container 27e6f776eca28eac3a949e115058559f022be8cda50b32a6906713f41eb46978: Status 404 returned error can't find the container with id 27e6f776eca28eac3a949e115058559f022be8cda50b32a6906713f41eb46978 Nov 01 08:42:51.682776 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:51.682711 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" event={"ID":"eec42fe9-a657-4b42-bb54-7849913616b4","Type":"ContainerStarted","Data":"27e6f776eca28eac3a949e115058559f022be8cda50b32a6906713f41eb46978"} Nov 01 08:42:51.694543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:51.694515 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj" Nov 01 08:42:57.170009 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:57.169975 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:42:57.170445 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:57.170161 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:42:57.686484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:57.686448 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:57.699659 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:57.699630 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:42:57.722664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:42:57.722634 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8" Nov 01 08:43:01.250073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:01.250038 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:01.250373 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:01.250110 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:01.250373 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:01.250137 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:01.733440 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:01.733393 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" event={"ID":"eec42fe9-a657-4b42-bb54-7849913616b4","Type":"ContainerStarted","Data":"969168af277773d76ab71cd9c38a07bdba698cdd64b438a55ed4a85d73008a3e"} Nov 01 08:43:01.897235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:01.897158 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podStartSLOduration=2.182900848 podStartE2EDuration="11.897133651s" totalImagesPullingTime="9.714232803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:01.895573165 +0000 UTC m=+604.983909841" watchObservedRunningTime="2026-11-01 08:43:01.897133651 +0000 UTC m=+604.985470324" Nov 01 08:43:02.098753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:02.098695 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:43:02.099183 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:02.099145 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:02.099383 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:02.099358 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:02.176609 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:02.176567 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:43:03.106251 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:03.106204 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:03.106688 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:03.106278 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:04.098210 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:04.098168 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:04.098511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:04.098476 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:05.100749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.098233 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:05.100749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.098309 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:05.893749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.887325 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5"] Nov 01 08:43:05.909420 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.909378 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.947605 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947567 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.947605 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947607 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.947921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947647 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.947921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947780 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.947921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947814 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.947921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947847 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.948101 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947958 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82d5\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.948101 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947982 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:05.948101 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.947997 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.011053 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:05.987569 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5"] Nov 01 08:43:06.048951 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.048917 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.048951 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.048955 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049213 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.048979 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049213 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049032 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049213 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049069 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049350 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049270 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049350 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049340 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82d5\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049450 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049374 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049450 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049401 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049463 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049505 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049780 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.049985 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.049963 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.050136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.050117 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.051920 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.051893 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.052109 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.052088 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.105267 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.104204 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:06.105267 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.104269 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:06.105865 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.105838 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:43:06.109215 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.109177 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.118486 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.114597 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82d5\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5\") pod \"maas-default-gateway-openshift-default-746ff8f49f-xk6c5\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.225652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.225553 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:06.698429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.688543 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5"] Nov 01 08:43:06.710445 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:43:06.710396 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31ff5cf_7c02_470b_b693_6a426ef895ec.slice/crio-e4f27217d6b47bf4982c4963f3f243e87529ded15c77ce2fdac9b8d32de3c091 WatchSource:0}: Error finding container e4f27217d6b47bf4982c4963f3f243e87529ded15c77ce2fdac9b8d32de3c091: Status 404 returned error can't find the container with id e4f27217d6b47bf4982c4963f3f243e87529ded15c77ce2fdac9b8d32de3c091 Nov 01 08:43:06.713876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.713835 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:06.713986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.713929 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:06.713986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.713970 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:06.764101 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:06.764074 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" event={"ID":"b31ff5cf-7c02-470b-b693-6a426ef895ec","Type":"ContainerStarted","Data":"e4f27217d6b47bf4982c4963f3f243e87529ded15c77ce2fdac9b8d32de3c091"} Nov 01 08:43:07.106294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:07.100186 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:07.106294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:07.100251 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:07.771060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:07.771025 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" event={"ID":"b31ff5cf-7c02-470b-b693-6a426ef895ec","Type":"ContainerStarted","Data":"566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846"} Nov 01 08:43:07.931732 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:07.931657 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podStartSLOduration=2.9316384859999998 podStartE2EDuration="2.931638486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:07.930437791 +0000 UTC m=+611.018774463" watchObservedRunningTime="2026-11-01 08:43:07.931638486 +0000 UTC m=+611.019975274" Nov 01 08:43:08.100270 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:08.100231 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:08.100632 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:08.100598 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:08.226328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:08.226275 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:08.228536 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:08.228503 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:08.228625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:08.228573 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:09.099760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:09.099209 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:09.099760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:09.099278 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:09.226740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:09.226664 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:09.227204 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:09.226782 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:10.097750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:10.097198 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 08:43:10.097750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:10.097271 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 08:43:10.226651 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:10.226601 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:10.226891 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:10.226664 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:11.226601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:11.226558 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:11.226814 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:11.226618 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:11.610141 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:11.610111 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:43:11.612521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:11.612492 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" Nov 01 08:43:12.226239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:12.226192 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:12.226403 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:12.226255 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:13.226837 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:13.226784 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:13.227356 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:13.226867 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:14.226279 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:14.226231 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 08:43:14.226477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:14.226315 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 08:43:15.481108 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:43:15.481063 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:43:16.202088 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:16.202051 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:16.202619 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:16.202593 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:16.203435 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:16.203413 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 08:43:19.274858 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.274820 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c65545f6c-spmfl"] Nov 01 08:43:19.279702 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.279679 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.376149 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.376102 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 08:43:19.376149 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.376109 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wlrzh\"" Nov 01 08:43:19.377730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.377695 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c65545f6c-spmfl"] Nov 01 08:43:19.404781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.404703 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gm5\" (UniqueName: \"kubernetes.io/projected/1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa-kube-api-access-l6gm5\") pod \"authorino-c65545f6c-spmfl\" (UID: \"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa\") " pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.404968 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.404875 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa-tls-cert\") pod \"authorino-c65545f6c-spmfl\" (UID: \"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa\") " pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.506118 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.506067 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa-tls-cert\") pod \"authorino-c65545f6c-spmfl\" (UID: \"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa\") " pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.506296 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.506186 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gm5\" (UniqueName: \"kubernetes.io/projected/1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa-kube-api-access-l6gm5\") pod \"authorino-c65545f6c-spmfl\" (UID: \"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa\") " pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.508845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.508811 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa-tls-cert\") pod \"authorino-c65545f6c-spmfl\" (UID: \"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa\") " pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.564349 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.564271 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gm5\" (UniqueName: \"kubernetes.io/projected/1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa-kube-api-access-l6gm5\") pod \"authorino-c65545f6c-spmfl\" (UID: \"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa\") " pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.592196 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.592153 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c65545f6c-spmfl" Nov 01 08:43:19.972573 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:19.972536 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c65545f6c-spmfl"] Nov 01 08:43:19.980953 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:43:19.980920 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b941aa0_8dbc_4dec_a4fa_2f8d5dafc3aa.slice/crio-f8e6d2bf37b1a7eb31fcdf2cae5c4135320cc9a23607582fd7809807a88da378 WatchSource:0}: Error finding container f8e6d2bf37b1a7eb31fcdf2cae5c4135320cc9a23607582fd7809807a88da378: Status 404 returned error can't find the container with id f8e6d2bf37b1a7eb31fcdf2cae5c4135320cc9a23607582fd7809807a88da378 Nov 01 08:43:20.857319 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:20.857285 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c65545f6c-spmfl" event={"ID":"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa","Type":"ContainerStarted","Data":"f8e6d2bf37b1a7eb31fcdf2cae5c4135320cc9a23607582fd7809807a88da378"} Nov 01 08:43:21.863285 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:21.863248 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c65545f6c-spmfl" event={"ID":"1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa","Type":"ContainerStarted","Data":"bdf1195c3d2d1229292c36945875cad52a0a68831334c9fde4b5d49ef5b36be3"} Nov 01 08:43:21.972275 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:21.972216 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c65545f6c-spmfl" podStartSLOduration=1.219996701 podStartE2EDuration="2.972200669s" totalImagesPullingTime="1.752203968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:21.97036917 +0000 UTC m=+625.058705841" watchObservedRunningTime="2026-11-01 08:43:21.972200669 +0000 UTC m=+625.060537340" Nov 01 08:43:45.476787 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:43:45.476680 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:43:49.844558 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.844523 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x"] Nov 01 08:43:49.848356 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.848328 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.922751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.922678 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.922751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.922751 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wx8m\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.922793 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.922908 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.922958 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.923009 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.923055 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.923108 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.923389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.923153 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:49.928870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.928832 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x"] Nov 01 08:43:49.949052 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:49.949020 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-685688c3-openshift-default-dockercfg-2spt9\"" Nov 01 08:43:50.024410 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024365 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024422 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024470 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024507 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024591 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024615 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx8m\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024645 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024685 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.024863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.024753 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.025341 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.025227 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.025341 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.025274 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.025526 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.025363 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.027885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.027857 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.028262 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.028216 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.028262 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.028232 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.028419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.028332 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.082751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.082687 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.083013 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.082987 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wx8m\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m\") pod \"e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.161369 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.161277 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:50.593632 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.593595 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x"] Nov 01 08:43:50.609885 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:43:50.609846 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244cff1b_680f_4c0b_a976_4e2127497800.slice/crio-d2f64c0ac951d806b43da7824452741435347e462ad35f2d3ac93249c4619262 WatchSource:0}: Error finding container d2f64c0ac951d806b43da7824452741435347e462ad35f2d3ac93249c4619262: Status 404 returned error can't find the container with id d2f64c0ac951d806b43da7824452741435347e462ad35f2d3ac93249c4619262 Nov 01 08:43:50.613138 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.613048 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:50.613257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.613190 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:50.613257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:50.613238 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:51.004247 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.004145 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" event={"ID":"244cff1b-680f-4c0b-a976-4e2127497800","Type":"ContainerStarted","Data":"64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc"} Nov 01 08:43:51.004772 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.004249 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" event={"ID":"244cff1b-680f-4c0b-a976-4e2127497800","Type":"ContainerStarted","Data":"d2f64c0ac951d806b43da7824452741435347e462ad35f2d3ac93249c4619262"} Nov 01 08:43:51.117767 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.117698 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podStartSLOduration=2.117684001 podStartE2EDuration="2.117684001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:51.116356317 +0000 UTC m=+654.204692988" watchObservedRunningTime="2026-11-01 08:43:51.117684001 +0000 UTC m=+654.206020727" Nov 01 08:43:51.161567 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.161536 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:51.167091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.167062 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:51.872262 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.872231 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p"] Nov 01 08:43:51.875883 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.875856 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.925337 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.925308 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5c25ae7a-openshift-default-dockercfg-nfd8w\"" Nov 01 08:43:51.954979 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.954948 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv76c\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955170 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955015 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955169 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955223 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955281 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955379 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955346 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955480 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955446 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955542 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955487 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.955542 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.955513 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:51.973772 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:51.973741 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p"] Nov 01 08:43:52.013678 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.013649 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:52.015332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.015310 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:43:52.056319 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056284 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056319 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056323 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056463 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv76c\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056550 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056661 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056617 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056818 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056795 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056895 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056845 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056944 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056896 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.056994 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.056946 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.057048 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.057026 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.057142 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.057121 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.057241 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.057217 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.057483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.057453 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.057662 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.057642 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.059181 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.059141 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.059950 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.059928 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.114111 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.114082 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.114332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.114308 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv76c\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c\") pod \"e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.190001 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.189907 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:52.619511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.619476 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p"] Nov 01 08:43:52.628695 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:43:52.628664 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36328be_0697_48b1_9ee3_9fc6970e9a8d.slice/crio-c0f7f9c5966d4f354bd13af6fbbf30dcdc3a77d58389ccb4d4ae52b66df70d99 WatchSource:0}: Error finding container c0f7f9c5966d4f354bd13af6fbbf30dcdc3a77d58389ccb4d4ae52b66df70d99: Status 404 returned error can't find the container with id c0f7f9c5966d4f354bd13af6fbbf30dcdc3a77d58389ccb4d4ae52b66df70d99 Nov 01 08:43:52.631150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.631111 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:52.631266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.631205 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:52.631266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:52.631237 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:43:53.020737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:53.020625 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" event={"ID":"a36328be-0697-48b1-9ee3-9fc6970e9a8d","Type":"ContainerStarted","Data":"c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3"} Nov 01 08:43:53.020737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:53.020687 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" event={"ID":"a36328be-0697-48b1-9ee3-9fc6970e9a8d","Type":"ContainerStarted","Data":"c0f7f9c5966d4f354bd13af6fbbf30dcdc3a77d58389ccb4d4ae52b66df70d99"} Nov 01 08:43:53.137654 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:53.137600 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podStartSLOduration=2.137584834 podStartE2EDuration="2.137584834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:53.136961083 +0000 UTC m=+656.225297755" watchObservedRunningTime="2026-11-01 08:43:53.137584834 +0000 UTC m=+656.225921506" Nov 01 08:43:53.190270 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:53.190233 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:53.195500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:53.195467 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:54.025293 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:54.025262 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:54.026292 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:54.026267 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:43:56.757896 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.757857 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p"] Nov 01 08:43:56.761784 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.761759 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:56.815974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.815940 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p"] Nov 01 08:43:56.857157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.857111 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-395735a2-dockercfg-mt9mt\"" Nov 01 08:43:56.857543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.857492 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-395735a2\"" Nov 01 08:43:56.910025 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.909985 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:56.910202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.910039 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9kxs\" (UniqueName: \"kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs\") pod \"payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:56.915990 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.915960 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw"] Nov 01 08:43:56.919632 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.919609 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:56.973495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:56.973465 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw"] Nov 01 08:43:57.011683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.011592 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9kxs\" (UniqueName: \"kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs\") pod \"payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:57.011683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.011653 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jzp\" (UniqueName: \"kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp\") pod \"payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.011985 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.011751 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume\") pod \"payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.011985 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.011843 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:57.064799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.064251 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-395735a2\"" Nov 01 08:43:57.070330 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.070280 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9kxs\" (UniqueName: \"kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs\") pod \"payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:57.072819 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.072773 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:57.112854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.112814 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jzp\" (UniqueName: \"kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp\") pod \"payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.113049 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.112894 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume\") pod \"payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.113623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.113604 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume\") pod \"payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.124339 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.124313 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-395735a2-dockercfg-mt9mt\"" Nov 01 08:43:57.126454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.126430 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:57.170946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.170911 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jzp\" (UniqueName: \"kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp\") pod \"payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.233481 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.233447 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:57.503134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.503098 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p"] Nov 01 08:43:57.510915 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:43:57.510877 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76da185_14cd_4937_846d_1f03f956a7df.slice/crio-3a164404f57256be114947faa913942f4af1413ca92defb531efd2969583bebc WatchSource:0}: Error finding container 3a164404f57256be114947faa913942f4af1413ca92defb531efd2969583bebc: Status 404 returned error can't find the container with id 3a164404f57256be114947faa913942f4af1413ca92defb531efd2969583bebc Nov 01 08:43:57.614609 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:57.614574 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw"] Nov 01 08:43:57.622580 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:43:57.622540 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59e7ada_4219_405a_ae17_e00cabacd12d.slice/crio-4da36f169914f5abf6d9da31fcef687652fcdbbfe0ad0222006c262ed2dd6c2b WatchSource:0}: Error finding container 4da36f169914f5abf6d9da31fcef687652fcdbbfe0ad0222006c262ed2dd6c2b: Status 404 returned error can't find the container with id 4da36f169914f5abf6d9da31fcef687652fcdbbfe0ad0222006c262ed2dd6c2b Nov 01 08:43:58.046045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.045345 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" event={"ID":"e59e7ada-4219-405a-ae17-e00cabacd12d","Type":"ContainerStarted","Data":"1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5"} Nov 01 08:43:58.046045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.045415 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" event={"ID":"e59e7ada-4219-405a-ae17-e00cabacd12d","Type":"ContainerStarted","Data":"4da36f169914f5abf6d9da31fcef687652fcdbbfe0ad0222006c262ed2dd6c2b"} Nov 01 08:43:58.046501 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.046353 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:43:58.048395 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.048365 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" event={"ID":"f76da185-14cd-4937-846d-1f03f956a7df","Type":"ContainerStarted","Data":"69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67"} Nov 01 08:43:58.048563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.048410 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" event={"ID":"f76da185-14cd-4937-846d-1f03f956a7df","Type":"ContainerStarted","Data":"3a164404f57256be114947faa913942f4af1413ca92defb531efd2969583bebc"} Nov 01 08:43:58.048563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.048432 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:43:58.155523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.155466 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" podStartSLOduration=2.155451262 podStartE2EDuration="2.155451262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:58.154225423 +0000 UTC m=+661.242562096" watchObservedRunningTime="2026-11-01 08:43:58.155451262 +0000 UTC m=+661.243787934" Nov 01 08:43:58.263589 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:58.263517 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" podStartSLOduration=2.263495596 podStartE2EDuration="2.263495596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:43:58.262190858 +0000 UTC m=+661.350527531" watchObservedRunningTime="2026-11-01 08:43:58.263495596 +0000 UTC m=+661.351832269" Nov 01 08:43:59.715242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.715208 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9"] Nov 01 08:43:59.719123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.719099 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.814439 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.814397 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-685688c3-serving-cert\"" Nov 01 08:43:59.814439 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.814417 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 08:43:59.814439 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.814413 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7zt8\"" Nov 01 08:43:59.814778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.814479 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 08:43:59.814778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.814491 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 08:43:59.816249 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.816220 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9"] Nov 01 08:43:59.849912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.849872 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.850118 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.850048 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw8kg\" (UniqueName: \"kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.850182 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.850148 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.850251 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.850233 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.951554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.951510 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.951805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.951590 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8kg\" (UniqueName: \"kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.951805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.951677 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.951805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.951779 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.952020 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:43:59.951820 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-serving-cert: secret "maas-api-e2e-ait-685688c3-serving-cert" not found Nov 01 08:43:59.952020 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:43:59.951914 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls podName:29101747-f539-4513-b93f-dd53c7b984d5 nodeName:}" failed. No retries permitted until 2026-11-01 08:44:00.451895469 +0000 UTC m=+663.540232131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls") pod "maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" (UID: "29101747-f539-4513-b93f-dd53c7b984d5") : secret "maas-api-e2e-ait-685688c3-serving-cert" not found Nov 01 08:43:59.952378 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.952339 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:43:59.954454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:43:59.954436 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:00.009828 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:00.009745 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8kg\" (UniqueName: \"kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:00.458186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:00.458152 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:00.460698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:00.460678 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\") pod \"maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:00.634137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:00.634098 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:01.015097 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:01.015065 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9"] Nov 01 08:44:01.027679 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:44:01.027634 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29101747_f539_4513_b93f_dd53c7b984d5.slice/crio-a5f7ad8fa49da2578593e64c32bf19548cf6ece810a0d17b4315b8771a7f842f WatchSource:0}: Error finding container a5f7ad8fa49da2578593e64c32bf19548cf6ece810a0d17b4315b8771a7f842f: Status 404 returned error can't find the container with id a5f7ad8fa49da2578593e64c32bf19548cf6ece810a0d17b4315b8771a7f842f Nov 01 08:44:01.029191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:01.029169 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 08:44:01.065392 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:01.065351 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" event={"ID":"29101747-f539-4513-b93f-dd53c7b984d5","Type":"ContainerStarted","Data":"a5f7ad8fa49da2578593e64c32bf19548cf6ece810a0d17b4315b8771a7f842f"} Nov 01 08:44:02.077827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:02.077792 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" event={"ID":"29101747-f539-4513-b93f-dd53c7b984d5","Type":"ContainerStarted","Data":"ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf"} Nov 01 08:44:02.078297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:02.077911 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:02.233303 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:02.233184 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" podStartSLOduration=2.299348501 podStartE2EDuration="3.233169174s" totalImagesPullingTime="933.820673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:02.232758367 +0000 UTC m=+665.321095038" watchObservedRunningTime="2026-11-01 08:44:02.233169174 +0000 UTC m=+665.321505846" Nov 01 08:44:07.270477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:07.270426 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:44:08.111708 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.111673 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:44:08.338446 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.338399 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42"] Nov 01 08:44:08.348763 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.348487 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.396806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.395490 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42"] Nov 01 08:44:08.407929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.407897 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5c25ae7a\"" Nov 01 08:44:08.408143 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.408046 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5c25ae7a-dockercfg-kkbzf\"" Nov 01 08:44:08.462962 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.462923 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume\") pod \"payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.462962 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.462965 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8lj\" (UniqueName: \"kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj\") pod \"payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.564288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.564240 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume\") pod \"payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.564288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.564291 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8lj\" (UniqueName: \"kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj\") pod \"payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.564993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.564974 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume\") pod \"payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.622884 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.622846 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8lj\" (UniqueName: \"kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj\") pod \"payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:08.665160 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:08.665061 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:09.057411 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:09.057336 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:44:09.066416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:09.066378 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42"] Nov 01 08:44:09.075188 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:44:09.075152 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590aad7a_00a5_4400_a80f_1a7d6020a11c.slice/crio-8a8200511edf25b6984534d5a67e52f638f4221e51f3f90cdb6a19452c39360e WatchSource:0}: Error finding container 8a8200511edf25b6984534d5a67e52f638f4221e51f3f90cdb6a19452c39360e: Status 404 returned error can't find the container with id 8a8200511edf25b6984534d5a67e52f638f4221e51f3f90cdb6a19452c39360e Nov 01 08:44:09.119854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:09.119822 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" event={"ID":"590aad7a-00a5-4400-a80f-1a7d6020a11c","Type":"ContainerStarted","Data":"8a8200511edf25b6984534d5a67e52f638f4221e51f3f90cdb6a19452c39360e"} Nov 01 08:44:10.063433 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.063407 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:44:10.127606 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.127567 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" event={"ID":"590aad7a-00a5-4400-a80f-1a7d6020a11c","Type":"ContainerStarted","Data":"333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11"} Nov 01 08:44:10.127870 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.127640 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:10.240357 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.240321 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj"] Nov 01 08:44:10.244230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.244203 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.294075 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.294036 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-395735a2-serving-cert\"" Nov 01 08:44:10.312476 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.312411 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" podStartSLOduration=2.312386301 podStartE2EDuration="2.312386301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:10.298144729 +0000 UTC m=+673.386481406" watchObservedRunningTime="2026-11-01 08:44:10.312386301 +0000 UTC m=+673.400722975" Nov 01 08:44:10.316681 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.316600 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj"] Nov 01 08:44:10.407239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.405542 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.407239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.406898 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.407239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.406964 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.407239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.407060 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ztk\" (UniqueName: \"kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.510380 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.510328 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.511108 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.511080 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.511888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.511864 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.516395 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.516366 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ztk\" (UniqueName: \"kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.526769 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:44:10.510878 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-serving-cert: secret "maas-api-e2e-ait-395735a2-serving-cert" not found Nov 01 08:44:10.526769 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:44:10.526046 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls podName:3aacead0-9203-47e9-a66d-be1ef5ab1e11 nodeName:}" failed. No retries permitted until 2026-11-01 08:44:11.02601058 +0000 UTC m=+674.114347253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls") pod "maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" (UID: "3aacead0-9203-47e9-a66d-be1ef5ab1e11") : secret "maas-api-e2e-ait-395735a2-serving-cert" not found Nov 01 08:44:10.526769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.516225 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.526769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.526677 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:10.599795 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:10.599612 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ztk\" (UniqueName: \"kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:11.032226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:11.032122 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:11.034840 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:11.034811 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\") pod \"maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:11.157710 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:11.157665 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:11.561426 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:11.561393 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj"] Nov 01 08:44:11.569069 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:44:11.569042 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aacead0_9203_47e9_a66d_be1ef5ab1e11.slice/crio-3918c5e718a6fe59c6d2dacd2230998f2d28805077f0ffbe416d7d09fa8d19ed WatchSource:0}: Error finding container 3918c5e718a6fe59c6d2dacd2230998f2d28805077f0ffbe416d7d09fa8d19ed: Status 404 returned error can't find the container with id 3918c5e718a6fe59c6d2dacd2230998f2d28805077f0ffbe416d7d09fa8d19ed Nov 01 08:44:12.139517 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:12.139482 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" event={"ID":"3aacead0-9203-47e9-a66d-be1ef5ab1e11","Type":"ContainerStarted","Data":"3918c5e718a6fe59c6d2dacd2230998f2d28805077f0ffbe416d7d09fa8d19ed"} Nov 01 08:44:13.146775 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:13.146705 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" event={"ID":"3aacead0-9203-47e9-a66d-be1ef5ab1e11","Type":"ContainerStarted","Data":"f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61"} Nov 01 08:44:13.147239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:13.146795 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:13.267765 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:13.267695 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" podStartSLOduration=2.331375407 podStartE2EDuration="3.26768035s" totalImagesPullingTime="936.304943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:13.265594249 +0000 UTC m=+676.353930920" watchObservedRunningTime="2026-11-01 08:44:13.26768035 +0000 UTC m=+676.356017023" Nov 01 08:44:15.477364 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:44:15.477329 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:44:15.764601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:15.764496 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:44:18.176014 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:18.175972 5710 generic.go:356] "Generic (PLEG): container finished" podID="244cff1b-680f-4c0b-a976-4e2127497800" containerID="64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc" exitCode=0 Nov 01 08:44:18.176467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:18.176049 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" event={"ID":"244cff1b-680f-4c0b-a976-4e2127497800","Type":"ContainerDied","Data":"64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc"} Nov 01 08:44:18.176514 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:18.176479 5710 scope.go:122] "RemoveContainer" containerID="64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc" Nov 01 08:44:18.177782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:18.177758 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:18.177852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:18.177824 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:18.177888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:18.177858 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:19.158173 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:19.158143 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:44:19.201744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:19.199976 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" event={"ID":"244cff1b-680f-4c0b-a976-4e2127497800","Type":"ContainerStarted","Data":"56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae"} Nov 01 08:44:20.162354 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:20.162311 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:44:20.163067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:20.163043 5710 patch_prober.go:29] interesting pod/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 08:44:20.163280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:20.163252 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 08:44:21.139338 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:21.139305 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:44:21.162470 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:21.162436 5710 patch_prober.go:29] interesting pod/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 08:44:21.162641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:21.162493 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 08:44:22.013996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:22.013954 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:44:22.161981 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:22.161947 5710 patch_prober.go:29] interesting pod/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 08:44:22.162363 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:22.162011 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 08:44:22.803436 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:22.803386 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:44:23.162202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:23.162136 5710 patch_prober.go:29] interesting pod/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 08:44:23.162762 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:23.162232 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 08:44:24.162311 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:24.162278 5710 patch_prober.go:29] interesting pod/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 08:44:24.162747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:24.162331 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 08:44:25.067978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.067915 5710 patch_prober.go:29] interesting pod/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:44:25.067978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.067980 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:44:25.166419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.166373 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:44:25.167868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.167846 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:44:25.329670 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.329582 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4"] Nov 01 08:44:25.341897 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.341866 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.390073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.390030 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4"] Nov 01 08:44:25.394494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.394462 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0f765cd6-serving-cert\"" Nov 01 08:44:25.434252 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.434212 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjjr\" (UniqueName: \"kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.434467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.434293 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.434537 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.434455 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.434638 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.434611 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.536230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.536186 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.536438 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.536275 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjjr\" (UniqueName: \"kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.536438 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.536338 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.536438 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.536371 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.536611 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:44:25.536456 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-serving-cert: secret "maas-api-e2e-derive-0f765cd6-serving-cert" not found Nov 01 08:44:25.536611 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:44:25.536534 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls podName:7e668c98-ba64-4454-ab7e-c10fbc65517d nodeName:}" failed. No retries permitted until 2026-11-01 08:44:26.036512705 +0000 UTC m=+689.124849371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls") pod "maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" (UID: "7e668c98-ba64-4454-ab7e-c10fbc65517d") : secret "maas-api-e2e-derive-0f765cd6-serving-cert" not found Nov 01 08:44:25.537145 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.537120 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.539004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.538986 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:25.597843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:25.597810 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjjr\" (UniqueName: \"kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:26.044756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.044596 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:26.047417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.047379 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\") pod \"maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:26.256954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.256917 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:26.267087 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.267055 5710 generic.go:356] "Generic (PLEG): container finished" podID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerID="c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3" exitCode=0 Nov 01 08:44:26.267087 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.267095 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" event={"ID":"a36328be-0697-48b1-9ee3-9fc6970e9a8d","Type":"ContainerDied","Data":"c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3"} Nov 01 08:44:26.267516 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.267503 5710 scope.go:122] "RemoveContainer" containerID="c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3" Nov 01 08:44:26.269055 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.269027 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:26.269159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.269101 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:26.269159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.269136 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:26.641084 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:26.641052 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4"] Nov 01 08:44:26.649571 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:44:26.649523 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e668c98_ba64_4454_ab7e_c10fbc65517d.slice/crio-07ada6e7d92ca48b5f0b8493d9d54d90dc0b29a200e5eb476dd5b89b8be07f69 WatchSource:0}: Error finding container 07ada6e7d92ca48b5f0b8493d9d54d90dc0b29a200e5eb476dd5b89b8be07f69: Status 404 returned error can't find the container with id 07ada6e7d92ca48b5f0b8493d9d54d90dc0b29a200e5eb476dd5b89b8be07f69 Nov 01 08:44:27.275471 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:27.275437 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" event={"ID":"a36328be-0697-48b1-9ee3-9fc6970e9a8d","Type":"ContainerStarted","Data":"497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1"} Nov 01 08:44:27.276960 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:27.276931 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" event={"ID":"7e668c98-ba64-4454-ab7e-c10fbc65517d","Type":"ContainerStarted","Data":"07ada6e7d92ca48b5f0b8493d9d54d90dc0b29a200e5eb476dd5b89b8be07f69"} Nov 01 08:44:28.195744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:28.191288 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:44:28.195744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:28.191442 5710 patch_prober.go:29] interesting pod/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 01 08:44:28.195744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:28.191488 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 01 08:44:28.319958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:28.319925 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" event={"ID":"7e668c98-ba64-4454-ab7e-c10fbc65517d","Type":"ContainerStarted","Data":"fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39"} Nov 01 08:44:28.320554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:28.320524 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:28.431050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:28.430978 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" podStartSLOduration=2.542954375 podStartE2EDuration="3.430957609s" totalImagesPullingTime="888.003234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:28.430610911 +0000 UTC m=+691.518947587" watchObservedRunningTime="2026-11-01 08:44:28.430957609 +0000 UTC m=+691.519294282" Nov 01 08:44:29.198744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:29.196184 5710 patch_prober.go:29] interesting pod/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 01 08:44:29.198744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:29.196255 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 01 08:44:30.199745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:30.192208 5710 patch_prober.go:29] interesting pod/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 01 08:44:30.199745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:30.192280 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 01 08:44:31.200745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:31.200190 5710 patch_prober.go:29] interesting pod/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 01 08:44:31.200745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:31.200276 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 01 08:44:32.199745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:32.199207 5710 patch_prober.go:29] interesting pod/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 01 08:44:32.199745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:32.199292 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 01 08:44:33.194931 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:33.194895 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:44:33.195444 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:33.195199 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:44:33.196144 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:33.196123 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:44:34.329458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:34.329427 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:44:40.458574 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:40.458537 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:44:42.126004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.125969 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs"] Nov 01 08:44:42.129920 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.129901 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.224966 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.224928 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0f765cd6-openshift-default-dockercfg-4fckl\"" Nov 01 08:44:42.227413 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.227382 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs"] Nov 01 08:44:42.242502 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242462 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l574w\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242578 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242611 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242671 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242631 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242697 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242771 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242880 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242863 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.242927 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242913 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.243000 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.242985 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.344523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344487 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l574w\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.344747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344541 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.344747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344564 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.344747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344589 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.344747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344614 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.344747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344646 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344755 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344795 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345182 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.344888 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345182 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.345153 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.345223 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.345070 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345366 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.345342 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.345493 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.345471 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.347184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.347152 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.347409 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.347390 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.402633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.402558 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.402821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.402751 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l574w\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w\") pod \"e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.443432 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.443393 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:42.875986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.875952 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs"] Nov 01 08:44:42.883485 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:44:42.883442 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d23143_aea5_4424_800e_9e94acd29808.slice/crio-f4817827f5c2b30e29b80aedaa25760bd970b79d9610c2ce65a4b9c8662dcc19 WatchSource:0}: Error finding container f4817827f5c2b30e29b80aedaa25760bd970b79d9610c2ce65a4b9c8662dcc19: Status 404 returned error can't find the container with id f4817827f5c2b30e29b80aedaa25760bd970b79d9610c2ce65a4b9c8662dcc19 Nov 01 08:44:42.886044 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.886007 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:42.886159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.886079 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:42.886159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:42.886111 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:44:43.409465 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:43.409426 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" event={"ID":"18d23143-aea5-4424-800e-9e94acd29808","Type":"ContainerStarted","Data":"3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85"} Nov 01 08:44:43.409940 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:43.409489 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" event={"ID":"18d23143-aea5-4424-800e-9e94acd29808","Type":"ContainerStarted","Data":"f4817827f5c2b30e29b80aedaa25760bd970b79d9610c2ce65a4b9c8662dcc19"} Nov 01 08:44:43.444079 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:43.444040 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:43.445914 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:43.445869 5710 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 08:44:43.446062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:43.445926 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 08:44:43.522847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:43.522787 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podStartSLOduration=1.522766901 podStartE2EDuration="1.522766901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:44:43.521788969 +0000 UTC m=+706.610125641" watchObservedRunningTime="2026-11-01 08:44:43.522766901 +0000 UTC m=+706.611103573" Nov 01 08:44:44.444653 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:44.444612 5710 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 08:44:44.445147 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:44.444671 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 08:44:45.444013 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:45.443973 5710 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 08:44:45.444218 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:45.444046 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 08:44:45.477564 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:44:45.477528 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:44:46.444128 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:46.444083 5710 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 08:44:46.444299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:46.444143 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 08:44:47.444839 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:47.444803 5710 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 08:44:47.445248 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:47.444907 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 08:44:48.444657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:48.444617 5710 patch_prober.go:29] interesting pod/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 08:44:48.444968 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:48.444668 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 08:44:49.452568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:49.452533 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:49.453069 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:49.452848 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:49.453770 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:49.453749 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:44:57.692423 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:44:57.692379 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42"] Nov 01 08:45:00.193614 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.193576 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h"] Nov 01 08:45:00.198964 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.198936 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.257995 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.257956 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbrm\" (UniqueName: \"kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm\") pod \"maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.258194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.258039 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.356186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.356155 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\"" Nov 01 08:45:00.359847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.359818 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbrm\" (UniqueName: \"kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm\") pod \"maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.359993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.359896 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.360613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.360592 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.381070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.381037 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h"] Nov 01 08:45:00.417584 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.417547 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbrm\" (UniqueName: \"kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm\") pod \"maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.515073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.514981 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:00.712531 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.712491 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9"] Nov 01 08:45:00.712752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.712729 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" podUID="29101747-f539-4513-b93f-dd53c7b984d5" containerName="maas-api" containerID="cri-o://ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf" gracePeriod=30 Nov 01 08:45:00.891220 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.891114 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h"] Nov 01 08:45:00.898003 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:00.897883 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fdd3b7_cdba_43e6_9c12_3e6b241571b4.slice/crio-44ee943960753e3433436dbe7733bf1b6a0308733697788a606c1bcc18d1088c WatchSource:0}: Error finding container 44ee943960753e3433436dbe7733bf1b6a0308733697788a606c1bcc18d1088c: Status 404 returned error can't find the container with id 44ee943960753e3433436dbe7733bf1b6a0308733697788a606c1bcc18d1088c Nov 01 08:45:00.988854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:00.988819 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:45:01.073765 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.073704 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\") pod \"29101747-f539-4513-b93f-dd53c7b984d5\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " Nov 01 08:45:01.073976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.073833 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg\" (UniqueName: \"kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg\") pod \"29101747-f539-4513-b93f-dd53c7b984d5\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " Nov 01 08:45:01.073976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.073913 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs\") pod \"29101747-f539-4513-b93f-dd53c7b984d5\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " Nov 01 08:45:01.074103 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.074026 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca\") pod \"29101747-f539-4513-b93f-dd53c7b984d5\" (UID: \"29101747-f539-4513-b93f-dd53c7b984d5\") " Nov 01 08:45:01.074395 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.074358 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca" pod "29101747-f539-4513-b93f-dd53c7b984d5" (UID: "29101747-f539-4513-b93f-dd53c7b984d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:01.074675 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.074437 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29101747-f539-4513-b93f-dd53c7b984d5-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:01.076098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.076069 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs" pod "29101747-f539-4513-b93f-dd53c7b984d5" (UID: "29101747-f539-4513-b93f-dd53c7b984d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:01.076098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.076088 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls" pod "29101747-f539-4513-b93f-dd53c7b984d5" (UID: "29101747-f539-4513-b93f-dd53c7b984d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:01.076098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.076088 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg" pod "29101747-f539-4513-b93f-dd53c7b984d5" (UID: "29101747-f539-4513-b93f-dd53c7b984d5"). InnerVolumeSpecName "kube-api-access-fw8kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:01.175813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.175766 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:01.175813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.175803 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw8kg\" (UniqueName: \"kubernetes.io/projected/29101747-f539-4513-b93f-dd53c7b984d5-kube-api-access-fw8kg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:01.175813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.175817 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29101747-f539-4513-b93f-dd53c7b984d5-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:01.514946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.514839 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerStarted","Data":"44ee943960753e3433436dbe7733bf1b6a0308733697788a606c1bcc18d1088c"} Nov 01 08:45:01.516242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.516212 5710 generic.go:356] "Generic (PLEG): container finished" podID="29101747-f539-4513-b93f-dd53c7b984d5" containerID="ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf" exitCode=0 Nov 01 08:45:01.516370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.516255 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" Nov 01 08:45:01.516370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.516287 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" event={"ID":"29101747-f539-4513-b93f-dd53c7b984d5","Type":"ContainerDied","Data":"ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf"} Nov 01 08:45:01.516370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.516334 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9" event={"ID":"29101747-f539-4513-b93f-dd53c7b984d5","Type":"ContainerDied","Data":"a5f7ad8fa49da2578593e64c32bf19548cf6ece810a0d17b4315b8771a7f842f"} Nov 01 08:45:01.516558 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.516411 5710 scope.go:122] "RemoveContainer" containerID="ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf" Nov 01 08:45:01.530419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.530248 5710 scope.go:122] "RemoveContainer" containerID="ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf" Nov 01 08:45:01.530757 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:01.530731 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf\": container with ID starting with ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf not found: ID does not exist" containerID="ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf" Nov 01 08:45:01.530828 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.530766 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf"} err="failed to get container status \"ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf\": rpc error: code = NotFound desc = could not find container \"ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf\": container with ID starting with ba89dbae0f1bd3e92447209d79e2146e0b85e894ed37f78ce31185523df34caf not found: ID does not exist" Nov 01 08:45:01.670746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.670700 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9"] Nov 01 08:45:01.674872 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:01.674825 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-685688c3-7f8f8bc7f8-cwcg9"] Nov 01 08:45:02.692827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:02.692762 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" podUID="590aad7a-00a5-4400-a80f-1a7d6020a11c" containerName="payload-processing" containerID="cri-o://333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11" gracePeriod=25 Nov 01 08:45:03.003497 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.003465 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:45:03.059502 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.059466 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29101747-f539-4513-b93f-dd53c7b984d5" path="/var/lib/kubelet/pods/29101747-f539-4513-b93f-dd53c7b984d5/volumes" Nov 01 08:45:03.068488 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.068459 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x"] Nov 01 08:45:03.068677 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.068656 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" containerID="cri-o://56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae" gracePeriod=30 Nov 01 08:45:03.099978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.099943 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj\" (UniqueName: \"kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj\") pod \"590aad7a-00a5-4400-a80f-1a7d6020a11c\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " Nov 01 08:45:03.100169 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.100011 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume\") pod \"590aad7a-00a5-4400-a80f-1a7d6020a11c\" (UID: \"590aad7a-00a5-4400-a80f-1a7d6020a11c\") " Nov 01 08:45:03.100439 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.100410 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume" pod "590aad7a-00a5-4400-a80f-1a7d6020a11c" (UID: "590aad7a-00a5-4400-a80f-1a7d6020a11c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:03.102392 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.102371 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj" pod "590aad7a-00a5-4400-a80f-1a7d6020a11c" (UID: "590aad7a-00a5-4400-a80f-1a7d6020a11c"). InnerVolumeSpecName "kube-api-access-kx8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:03.153124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.153082 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj"] Nov 01 08:45:03.153620 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.153564 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" podUID="3aacead0-9203-47e9-a66d-be1ef5ab1e11" containerName="maas-api" containerID="cri-o://f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61" gracePeriod=30 Nov 01 08:45:03.201601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.201571 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/590aad7a-00a5-4400-a80f-1a7d6020a11c-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:03.201601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.201598 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8lj\" (UniqueName: \"kubernetes.io/projected/590aad7a-00a5-4400-a80f-1a7d6020a11c-kube-api-access-kx8lj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:03.448437 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.448411 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:45:03.505844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.505796 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca\") pod \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " Nov 01 08:45:03.505844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.505865 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\") pod \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " Nov 01 08:45:03.506091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.505901 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk\" (UniqueName: \"kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk\") pod \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " Nov 01 08:45:03.506091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.506012 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs\") pod \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\" (UID: \"3aacead0-9203-47e9-a66d-be1ef5ab1e11\") " Nov 01 08:45:03.506220 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.506194 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca" pod "3aacead0-9203-47e9-a66d-be1ef5ab1e11" (UID: "3aacead0-9203-47e9-a66d-be1ef5ab1e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:03.506467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.506444 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aacead0-9203-47e9-a66d-be1ef5ab1e11-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:03.508386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.508338 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs" pod "3aacead0-9203-47e9-a66d-be1ef5ab1e11" (UID: "3aacead0-9203-47e9-a66d-be1ef5ab1e11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:03.508386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.508359 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk" pod "3aacead0-9203-47e9-a66d-be1ef5ab1e11" (UID: "3aacead0-9203-47e9-a66d-be1ef5ab1e11"). InnerVolumeSpecName "kube-api-access-j6ztk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:03.508386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.508371 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls" pod "3aacead0-9203-47e9-a66d-be1ef5ab1e11" (UID: "3aacead0-9203-47e9-a66d-be1ef5ab1e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:03.544859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.544756 5710 generic.go:356] "Generic (PLEG): container finished" podID="590aad7a-00a5-4400-a80f-1a7d6020a11c" containerID="333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11" exitCode=0 Nov 01 08:45:03.544859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.544811 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" Nov 01 08:45:03.545117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.544856 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" event={"ID":"590aad7a-00a5-4400-a80f-1a7d6020a11c","Type":"ContainerDied","Data":"333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11"} Nov 01 08:45:03.545117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.544946 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42" event={"ID":"590aad7a-00a5-4400-a80f-1a7d6020a11c","Type":"ContainerDied","Data":"8a8200511edf25b6984534d5a67e52f638f4221e51f3f90cdb6a19452c39360e"} Nov 01 08:45:03.545117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.545003 5710 scope.go:122] "RemoveContainer" containerID="333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11" Nov 01 08:45:03.547882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.547852 5710 generic.go:356] "Generic (PLEG): container finished" podID="3aacead0-9203-47e9-a66d-be1ef5ab1e11" containerID="f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61" exitCode=0 Nov 01 08:45:03.548047 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.547929 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" Nov 01 08:45:03.548047 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.547934 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" event={"ID":"3aacead0-9203-47e9-a66d-be1ef5ab1e11","Type":"ContainerDied","Data":"f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61"} Nov 01 08:45:03.548047 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.548004 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj" event={"ID":"3aacead0-9203-47e9-a66d-be1ef5ab1e11","Type":"ContainerDied","Data":"3918c5e718a6fe59c6d2dacd2230998f2d28805077f0ffbe416d7d09fa8d19ed"} Nov 01 08:45:03.556263 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.556240 5710 scope.go:122] "RemoveContainer" containerID="333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11" Nov 01 08:45:03.556707 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:03.556680 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11\": container with ID starting with 333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11 not found: ID does not exist" containerID="333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11" Nov 01 08:45:03.556869 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.556757 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11"} err="failed to get container status \"333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11\": rpc error: code = NotFound desc = could not find container \"333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11\": container with ID starting with 333cca9115a1fea3c01186c2282ac44d8cde37dae89af9f972459b7cfffbdc11 not found: ID does not exist" Nov 01 08:45:03.556869 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.556781 5710 scope.go:122] "RemoveContainer" containerID="f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61" Nov 01 08:45:03.569152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.569127 5710 scope.go:122] "RemoveContainer" containerID="f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61" Nov 01 08:45:03.569705 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:03.569678 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61\": container with ID starting with f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61 not found: ID does not exist" containerID="f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61" Nov 01 08:45:03.569865 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.569804 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61"} err="failed to get container status \"f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61\": rpc error: code = NotFound desc = could not find container \"f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61\": container with ID starting with f67f04f9a9dfa20ce32141234ed59ec9b1f592071ee4083110101dc6c741df61 not found: ID does not exist" Nov 01 08:45:03.597935 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.597903 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw"] Nov 01 08:45:03.607685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.607651 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6ztk\" (UniqueName: \"kubernetes.io/projected/3aacead0-9203-47e9-a66d-be1ef5ab1e11-kube-api-access-j6ztk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:03.607685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.607683 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:03.607685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.607698 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aacead0-9203-47e9-a66d-be1ef5ab1e11-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:03.665319 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.665281 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj"] Nov 01 08:45:03.669423 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.669398 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-395735a2-58758bbcd6-nk6nj"] Nov 01 08:45:03.690075 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.690038 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p"] Nov 01 08:45:03.772174 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.772138 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42"] Nov 01 08:45:03.783860 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:03.783831 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5c25ae7a-7b7468f67c-p4f42"] Nov 01 08:45:04.714571 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:04.714485 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p"] Nov 01 08:45:04.715557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:04.715520 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" containerID="cri-o://497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1" gracePeriod=30 Nov 01 08:45:05.058122 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:05.058041 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aacead0-9203-47e9-a66d-be1ef5ab1e11" path="/var/lib/kubelet/pods/3aacead0-9203-47e9-a66d-be1ef5ab1e11/volumes" Nov 01 08:45:05.058505 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:05.058427 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590aad7a-00a5-4400-a80f-1a7d6020a11c" path="/var/lib/kubelet/pods/590aad7a-00a5-4400-a80f-1a7d6020a11c/volumes" Nov 01 08:45:06.313766 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.313731 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6"] Nov 01 08:45:06.314211 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314197 5710 cpu_manager.go:414] "containerMap: removing container" podUID="590aad7a-00a5-4400-a80f-1a7d6020a11c" containerName="payload-processing" Nov 01 08:45:06.314268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314212 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590aad7a-00a5-4400-a80f-1a7d6020a11c" containerName="payload-processing" Nov 01 08:45:06.314268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314238 5710 cpu_manager.go:414] "containerMap: removing container" podUID="29101747-f539-4513-b93f-dd53c7b984d5" containerName="maas-api" Nov 01 08:45:06.314268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314244 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29101747-f539-4513-b93f-dd53c7b984d5" containerName="maas-api" Nov 01 08:45:06.314268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314257 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3aacead0-9203-47e9-a66d-be1ef5ab1e11" containerName="maas-api" Nov 01 08:45:06.314268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314262 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aacead0-9203-47e9-a66d-be1ef5ab1e11" containerName="maas-api" Nov 01 08:45:06.314456 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314353 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="29101747-f539-4513-b93f-dd53c7b984d5" containerName="maas-api" Nov 01 08:45:06.314456 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314362 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aacead0-9203-47e9-a66d-be1ef5ab1e11" containerName="maas-api" Nov 01 08:45:06.314456 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.314372 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="590aad7a-00a5-4400-a80f-1a7d6020a11c" containerName="payload-processing" Nov 01 08:45:06.321165 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.321142 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.373697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.373658 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6"] Nov 01 08:45:06.417317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.417283 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 08:45:06.449129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.449087 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.449321 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.449232 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8hb\" (UniqueName: \"kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.449377 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.449326 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.550504 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.550461 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.550760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.550536 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8hb\" (UniqueName: \"kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.550760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.550584 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.551274 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.551246 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.553148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.553118 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.627345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.627308 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8hb\" (UniqueName: \"kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb\") pod \"maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:06.635774 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:06.635743 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:07.020455 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:07.020373 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6"] Nov 01 08:45:07.028594 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:07.028552 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa78d778_0554_4cb5_9a10_261499cb73f6.slice/crio-c33e02244f224a280901bf25d3668efdc8ae4f0d8d200a700106a03ad8978287 WatchSource:0}: Error finding container c33e02244f224a280901bf25d3668efdc8ae4f0d8d200a700106a03ad8978287: Status 404 returned error can't find the container with id c33e02244f224a280901bf25d3668efdc8ae4f0d8d200a700106a03ad8978287 Nov 01 08:45:07.568954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:07.568918 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:45:07.602005 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:07.601957 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" event={"ID":"fa78d778-0554-4cb5-9a10-261499cb73f6","Type":"ContainerStarted","Data":"c33e02244f224a280901bf25d3668efdc8ae4f0d8d200a700106a03ad8978287"} Nov 01 08:45:08.328618 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.328588 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:45:08.379697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379659 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.379918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379740 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.379918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379770 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.379918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379798 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.379918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379849 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.379918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379906 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.380233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379968 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.380233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379976 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:08.380233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.379991 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.380233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.380057 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert\") pod \"244cff1b-680f-4c0b-a976-4e2127497800\" (UID: \"244cff1b-680f-4c0b-a976-4e2127497800\") " Nov 01 08:45:08.380435 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.380395 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:08.380490 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.380461 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.380545 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.380487 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:08.380545 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.380484 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:08.380545 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.380514 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:08.382299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.382255 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:08.382425 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.382355 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 08:45:08.382789 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.382765 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:08.382908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.382794 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m" pod "244cff1b-680f-4c0b-a976-4e2127497800" (UID: "244cff1b-680f-4c0b-a976-4e2127497800"). InnerVolumeSpecName "kube-api-access-6wx8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:08.481184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481142 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481180 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481193 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481208 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481222 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244cff1b-680f-4c0b-a976-4e2127497800-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481234 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244cff1b-680f-4c0b-a976-4e2127497800-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481247 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244cff1b-680f-4c0b-a976-4e2127497800-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.481400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.481259 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wx8m\" (UniqueName: \"kubernetes.io/projected/244cff1b-680f-4c0b-a976-4e2127497800-kube-api-access-6wx8m\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:08.598961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.598917 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" podUID="e59e7ada-4219-405a-ae17-e00cabacd12d" containerName="payload-processing" containerID="cri-o://1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5" gracePeriod=25 Nov 01 08:45:08.609376 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.609345 5710 generic.go:356] "Generic (PLEG): container finished" podID="244cff1b-680f-4c0b-a976-4e2127497800" containerID="56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae" exitCode=0 Nov 01 08:45:08.609550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.609422 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" Nov 01 08:45:08.609550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.609424 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" event={"ID":"244cff1b-680f-4c0b-a976-4e2127497800","Type":"ContainerDied","Data":"56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae"} Nov 01 08:45:08.609550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.609485 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x" event={"ID":"244cff1b-680f-4c0b-a976-4e2127497800","Type":"ContainerDied","Data":"d2f64c0ac951d806b43da7824452741435347e462ad35f2d3ac93249c4619262"} Nov 01 08:45:08.609550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.609516 5710 scope.go:122] "RemoveContainer" containerID="56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae" Nov 01 08:45:08.611789 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.611757 5710 generic.go:356] "Generic (PLEG): container finished" podID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerID="c39a15010c020a3a5d94ec0d229251c6d7e3494091ae983f149d23f0db09126d" exitCode=60 Nov 01 08:45:08.611949 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.611813 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerDied","Data":"c39a15010c020a3a5d94ec0d229251c6d7e3494091ae983f149d23f0db09126d"} Nov 01 08:45:08.612100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.612084 5710 scope.go:122] "RemoveContainer" containerID="c39a15010c020a3a5d94ec0d229251c6d7e3494091ae983f149d23f0db09126d" Nov 01 08:45:08.690495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.690457 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" podUID="f76da185-14cd-4937-846d-1f03f956a7df" containerName="payload-pre-processing" containerID="cri-o://69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67" gracePeriod=25 Nov 01 08:45:08.768952 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.768933 5710 scope.go:122] "RemoveContainer" containerID="64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc" Nov 01 08:45:08.802890 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.802866 5710 scope.go:122] "RemoveContainer" containerID="56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae" Nov 01 08:45:08.803214 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:08.803192 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae\": container with ID starting with 56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae not found: ID does not exist" containerID="56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae" Nov 01 08:45:08.803299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.803228 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae"} err="failed to get container status \"56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae\": rpc error: code = NotFound desc = could not find container \"56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae\": container with ID starting with 56ebb271d5d77673cc148fb6b50aec026954bbd7d77f376688f1c96b70c46aae not found: ID does not exist" Nov 01 08:45:08.803299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.803254 5710 scope.go:122] "RemoveContainer" containerID="64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc" Nov 01 08:45:08.803599 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:08.803535 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc\": container with ID starting with 64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc not found: ID does not exist" containerID="64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc" Nov 01 08:45:08.803599 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.803561 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc"} err="failed to get container status \"64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc\": rpc error: code = NotFound desc = could not find container \"64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc\": container with ID starting with 64f779de4b15488aa98c96030b6657da12938c53a75100e284122e4b74e2c8cc not found: ID does not exist" Nov 01 08:45:08.897097 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.896991 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x"] Nov 01 08:45:08.905978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:08.900587 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-685688c3-openshift-default-5bdb777bd7-sng4x"] Nov 01 08:45:09.045641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.045609 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:45:09.048203 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.048176 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:45:09.057921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.057884 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244cff1b-680f-4c0b-a976-4e2127497800" path="/var/lib/kubelet/pods/244cff1b-680f-4c0b-a976-4e2127497800/volumes" Nov 01 08:45:09.193080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.190182 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp\" (UniqueName: \"kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp\") pod \"e59e7ada-4219-405a-ae17-e00cabacd12d\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " Nov 01 08:45:09.193080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.190247 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume\") pod \"e59e7ada-4219-405a-ae17-e00cabacd12d\" (UID: \"e59e7ada-4219-405a-ae17-e00cabacd12d\") " Nov 01 08:45:09.193080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.190287 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs\" (UniqueName: \"kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs\") pod \"f76da185-14cd-4937-846d-1f03f956a7df\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " Nov 01 08:45:09.193080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.190372 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume\") pod \"f76da185-14cd-4937-846d-1f03f956a7df\" (UID: \"f76da185-14cd-4937-846d-1f03f956a7df\") " Nov 01 08:45:09.193080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.191276 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume" pod "f76da185-14cd-4937-846d-1f03f956a7df" (UID: "f76da185-14cd-4937-846d-1f03f956a7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:09.193080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.191704 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume" pod "e59e7ada-4219-405a-ae17-e00cabacd12d" (UID: "e59e7ada-4219-405a-ae17-e00cabacd12d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:09.194368 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.194340 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp" pod "e59e7ada-4219-405a-ae17-e00cabacd12d" (UID: "e59e7ada-4219-405a-ae17-e00cabacd12d"). InnerVolumeSpecName "kube-api-access-h6jzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:09.197866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.197839 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs" pod "f76da185-14cd-4937-846d-1f03f956a7df" (UID: "f76da185-14cd-4937-846d-1f03f956a7df"). InnerVolumeSpecName "kube-api-access-t9kxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:09.291770 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.291728 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jzp\" (UniqueName: \"kubernetes.io/projected/e59e7ada-4219-405a-ae17-e00cabacd12d-kube-api-access-h6jzp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:09.291770 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.291758 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59e7ada-4219-405a-ae17-e00cabacd12d-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:09.291770 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.291769 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9kxs\" (UniqueName: \"kubernetes.io/projected/f76da185-14cd-4937-846d-1f03f956a7df-kube-api-access-t9kxs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:09.291770 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.291779 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76da185-14cd-4937-846d-1f03f956a7df-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:09.620287 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.620247 5710 generic.go:356] "Generic (PLEG): container finished" podID="e59e7ada-4219-405a-ae17-e00cabacd12d" containerID="1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5" exitCode=0 Nov 01 08:45:09.620832 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.620302 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" Nov 01 08:45:09.620832 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.620297 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" event={"ID":"e59e7ada-4219-405a-ae17-e00cabacd12d","Type":"ContainerDied","Data":"1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5"} Nov 01 08:45:09.620832 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.620450 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw" event={"ID":"e59e7ada-4219-405a-ae17-e00cabacd12d","Type":"ContainerDied","Data":"4da36f169914f5abf6d9da31fcef687652fcdbbfe0ad0222006c262ed2dd6c2b"} Nov 01 08:45:09.620832 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.620488 5710 scope.go:122] "RemoveContainer" containerID="1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5" Nov 01 08:45:09.623150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.623117 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerStarted","Data":"6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3"} Nov 01 08:45:09.627144 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.627112 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" event={"ID":"fa78d778-0554-4cb5-9a10-261499cb73f6","Type":"ContainerStarted","Data":"896b5ad5ef1242a1b098845f1ab67c048f70c78a1eb4e5f6f8f970929a87155e"} Nov 01 08:45:09.629229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.629201 5710 generic.go:356] "Generic (PLEG): container finished" podID="f76da185-14cd-4937-846d-1f03f956a7df" containerID="69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67" exitCode=0 Nov 01 08:45:09.629343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.629255 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" Nov 01 08:45:09.629408 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.629377 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" event={"ID":"f76da185-14cd-4937-846d-1f03f956a7df","Type":"ContainerDied","Data":"69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67"} Nov 01 08:45:09.629457 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.629427 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p" event={"ID":"f76da185-14cd-4937-846d-1f03f956a7df","Type":"ContainerDied","Data":"3a164404f57256be114947faa913942f4af1413ca92defb531efd2969583bebc"} Nov 01 08:45:09.633358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.633336 5710 scope.go:122] "RemoveContainer" containerID="1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5" Nov 01 08:45:09.633709 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:09.633675 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5\": container with ID starting with 1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5 not found: ID does not exist" containerID="1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5" Nov 01 08:45:09.634542 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.633735 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5"} err="failed to get container status \"1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5\": rpc error: code = NotFound desc = could not find container \"1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5\": container with ID starting with 1f6385fd0625179de763e94abc8837415baa23b13b48697afcfb7fed86513ea5 not found: ID does not exist" Nov 01 08:45:09.634542 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.633759 5710 scope.go:122] "RemoveContainer" containerID="69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67" Nov 01 08:45:09.643547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.643524 5710 scope.go:122] "RemoveContainer" containerID="69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67" Nov 01 08:45:09.643915 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:09.643894 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67\": container with ID starting with 69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67 not found: ID does not exist" containerID="69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67" Nov 01 08:45:09.643999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.643924 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67"} err="failed to get container status \"69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67\": rpc error: code = NotFound desc = could not find container \"69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67\": container with ID starting with 69c257d43b2f044550ab408a26a23fa19b02b7de141e25a90f1e6ba31c3b5d67 not found: ID does not exist" Nov 01 08:45:09.773876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.773842 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p"] Nov 01 08:45:09.779688 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.779658 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-395735a2-6d47c484c-kjg9p"] Nov 01 08:45:09.892700 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.890340 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" podStartSLOduration=1.9880157 podStartE2EDuration="3.890316152s" totalImagesPullingTime="1.902300452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:09.886472109 +0000 UTC m=+732.974808784" watchObservedRunningTime="2026-11-01 08:45:09.890316152 +0000 UTC m=+732.978652826" Nov 01 08:45:09.992953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.992918 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw"] Nov 01 08:45:09.999873 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:09.999844 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-395735a2-cbfbb885c-64wfw"] Nov 01 08:45:10.028034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.027960 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:45:10.100371 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100331 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100376 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100434 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100475 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100513 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100626 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100688 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.100843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100694 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:10.100843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100778 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.101050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100818 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:10.101050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100848 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo\") pod \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\" (UID: \"a36328be-0697-48b1-9ee3-9fc6970e9a8d\") " Nov 01 08:45:10.101050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100989 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:10.101050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.100999 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:10.101050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.101007 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:10.101404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.101231 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.101404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.101245 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.101404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.101254 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.101404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.101268 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.101404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.101283 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.102976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.102940 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:10.103076 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.103005 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "kube-api-access-dv76c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:10.103123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.103109 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 08:45:10.103228 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.103208 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token" pod "a36328be-0697-48b1-9ee3-9fc6970e9a8d" (UID: "a36328be-0697-48b1-9ee3-9fc6970e9a8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:10.108388 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.108340 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" podStartSLOduration=3.34639158 podStartE2EDuration="10.108327259s" totalImagesPullingTime="6.761935679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:10.106816915 +0000 UTC m=+733.195153588" watchObservedRunningTime="2026-11-01 08:45:10.108327259 +0000 UTC m=+733.196663930" Nov 01 08:45:10.202916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.202832 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.202916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.202857 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.202916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.202866 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36328be-0697-48b1-9ee3-9fc6970e9a8d-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.202916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.202876 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv76c\" (UniqueName: \"kubernetes.io/projected/a36328be-0697-48b1-9ee3-9fc6970e9a8d-kube-api-access-dv76c\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:10.636955 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.636924 5710 generic.go:356] "Generic (PLEG): container finished" podID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerID="497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1" exitCode=0 Nov 01 08:45:10.637498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.636974 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" Nov 01 08:45:10.637498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.637008 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" event={"ID":"a36328be-0697-48b1-9ee3-9fc6970e9a8d","Type":"ContainerDied","Data":"497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1"} Nov 01 08:45:10.637498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.637060 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p" event={"ID":"a36328be-0697-48b1-9ee3-9fc6970e9a8d","Type":"ContainerDied","Data":"c0f7f9c5966d4f354bd13af6fbbf30dcdc3a77d58389ccb4d4ae52b66df70d99"} Nov 01 08:45:10.637498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.637082 5710 scope.go:122] "RemoveContainer" containerID="497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1" Nov 01 08:45:10.640306 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.640277 5710 generic.go:356] "Generic (PLEG): container finished" podID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerID="6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3" exitCode=60 Nov 01 08:45:10.640463 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.640368 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerDied","Data":"6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3"} Nov 01 08:45:10.640623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.640592 5710 scope.go:122] "RemoveContainer" containerID="6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3" Nov 01 08:45:10.640935 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:10.640917 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h_odh-ai-gateway-infra(67fdd3b7-cdba-43e6-9c12-3e6b241571b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" Nov 01 08:45:10.642163 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.642138 5710 generic.go:356] "Generic (PLEG): container finished" podID="fa78d778-0554-4cb5-9a10-261499cb73f6" containerID="896b5ad5ef1242a1b098845f1ab67c048f70c78a1eb4e5f6f8f970929a87155e" exitCode=0 Nov 01 08:45:10.642253 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.642178 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" event={"ID":"fa78d778-0554-4cb5-9a10-261499cb73f6","Type":"ContainerDied","Data":"896b5ad5ef1242a1b098845f1ab67c048f70c78a1eb4e5f6f8f970929a87155e"} Nov 01 08:45:10.647888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.647869 5710 scope.go:122] "RemoveContainer" containerID="c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3" Nov 01 08:45:10.657152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.657130 5710 scope.go:122] "RemoveContainer" containerID="497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1" Nov 01 08:45:10.657435 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:10.657416 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1\": container with ID starting with 497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1 not found: ID does not exist" containerID="497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1" Nov 01 08:45:10.657509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.657444 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1"} err="failed to get container status \"497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1\": rpc error: code = NotFound desc = could not find container \"497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1\": container with ID starting with 497c05d7c264ba66cda842977d9f2c20ffe11f1df0cd2d308027d8e292c355a1 not found: ID does not exist" Nov 01 08:45:10.657509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.657462 5710 scope.go:122] "RemoveContainer" containerID="c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3" Nov 01 08:45:10.657704 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:10.657689 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3\": container with ID starting with c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3 not found: ID does not exist" containerID="c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3" Nov 01 08:45:10.657780 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.657710 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3"} err="failed to get container status \"c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3\": rpc error: code = NotFound desc = could not find container \"c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3\": container with ID starting with c7774a40e5f43669fde2891152fa1867f280965992002abeb2e00e8613ee7ab3 not found: ID does not exist" Nov 01 08:45:10.657780 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.657744 5710 scope.go:122] "RemoveContainer" containerID="c39a15010c020a3a5d94ec0d229251c6d7e3494091ae983f149d23f0db09126d" Nov 01 08:45:10.791436 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.791403 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p"] Nov 01 08:45:10.795239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:10.795215 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5c25ae7a-openshift-default-865848bf79-ghj2p"] Nov 01 08:45:11.065105 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.065023 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" path="/var/lib/kubelet/pods/a36328be-0697-48b1-9ee3-9fc6970e9a8d/volumes" Nov 01 08:45:11.065465 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.065450 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59e7ada-4219-405a-ae17-e00cabacd12d" path="/var/lib/kubelet/pods/e59e7ada-4219-405a-ae17-e00cabacd12d/volumes" Nov 01 08:45:11.065853 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.065837 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76da185-14cd-4937-846d-1f03f956a7df" path="/var/lib/kubelet/pods/f76da185-14cd-4937-846d-1f03f956a7df/volumes" Nov 01 08:45:11.254203 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254163 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw"] Nov 01 08:45:11.254682 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254667 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" Nov 01 08:45:11.254740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254684 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" Nov 01 08:45:11.254740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254700 5710 cpu_manager.go:414] "containerMap: removing container" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" Nov 01 08:45:11.254740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254704 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254748 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f76da185-14cd-4937-846d-1f03f956a7df" containerName="payload-pre-processing" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254754 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76da185-14cd-4937-846d-1f03f956a7df" containerName="payload-pre-processing" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254772 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e59e7ada-4219-405a-ae17-e00cabacd12d" containerName="payload-processing" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254777 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59e7ada-4219-405a-ae17-e00cabacd12d" containerName="payload-processing" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254848 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76da185-14cd-4937-846d-1f03f956a7df" containerName="payload-pre-processing" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254856 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254864 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254869 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59e7ada-4219-405a-ae17-e00cabacd12d" containerName="payload-processing" Nov 01 08:45:11.254871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.254877 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" Nov 01 08:45:11.259747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.259686 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.309170 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.309137 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b711ae-openshift-default-dockercfg-ms66n\"" Nov 01 08:45:11.313997 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.313954 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314014 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314045 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314088 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314176 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbs6\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314241 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314280 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314323 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.314453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.314355 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.316890 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.316808 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw"] Nov 01 08:45:11.415090 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415053 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbs6\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415135 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415183 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415221 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415254 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415304 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415336 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415368 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.415520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.415414 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.416810 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.416385 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.417821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.417789 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.418962 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.418931 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.419226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.419195 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.419330 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.419283 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.422134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.421938 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.422249 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.422151 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.474730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.474682 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.474986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.474964 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbs6\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.572998 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.572898 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:11.666324 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.665334 5710 scope.go:122] "RemoveContainer" containerID="6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3" Nov 01 08:45:11.666324 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:11.665601 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h_odh-ai-gateway-infra(67fdd3b7-cdba-43e6-9c12-3e6b241571b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" Nov 01 08:45:11.826658 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.826587 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:11.920340 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.920310 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca\") pod \"fa78d778-0554-4cb5-9a10-261499cb73f6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " Nov 01 08:45:11.920529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.920441 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb\" (UniqueName: \"kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb\") pod \"fa78d778-0554-4cb5-9a10-261499cb73f6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " Nov 01 08:45:11.920529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.920479 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca\") pod \"fa78d778-0554-4cb5-9a10-261499cb73f6\" (UID: \"fa78d778-0554-4cb5-9a10-261499cb73f6\") " Nov 01 08:45:11.920834 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.920803 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca" pod "fa78d778-0554-4cb5-9a10-261499cb73f6" (UID: "fa78d778-0554-4cb5-9a10-261499cb73f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:11.920928 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.920904 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:11.922889 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.922863 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca" pod "fa78d778-0554-4cb5-9a10-261499cb73f6" (UID: "fa78d778-0554-4cb5-9a10-261499cb73f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:11.922889 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:11.922879 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb" pod "fa78d778-0554-4cb5-9a10-261499cb73f6" (UID: "fa78d778-0554-4cb5-9a10-261499cb73f6"). InnerVolumeSpecName "kube-api-access-8h8hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:12.002738 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.002680 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw"] Nov 01 08:45:12.010248 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:12.010197 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dde983_43b7_4c73_ae19_4955151dbf81.slice/crio-0d7dffcd19018429806f7026dc200f4025c72f20a8e5f87e01ebdc433fecb348 WatchSource:0}: Error finding container 0d7dffcd19018429806f7026dc200f4025c72f20a8e5f87e01ebdc433fecb348: Status 404 returned error can't find the container with id 0d7dffcd19018429806f7026dc200f4025c72f20a8e5f87e01ebdc433fecb348 Nov 01 08:45:12.012654 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.012619 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:12.012767 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.012752 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:12.012812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.012795 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:12.022294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.022261 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h8hb\" (UniqueName: \"kubernetes.io/projected/fa78d778-0554-4cb5-9a10-261499cb73f6-kube-api-access-8h8hb\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:12.022294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.022296 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa78d778-0554-4cb5-9a10-261499cb73f6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:12.672317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.672288 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" Nov 01 08:45:12.672795 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.672330 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6" event={"ID":"fa78d778-0554-4cb5-9a10-261499cb73f6","Type":"ContainerDied","Data":"c33e02244f224a280901bf25d3668efdc8ae4f0d8d200a700106a03ad8978287"} Nov 01 08:45:12.672795 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.672395 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33e02244f224a280901bf25d3668efdc8ae4f0d8d200a700106a03ad8978287" Nov 01 08:45:12.674235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.674207 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" event={"ID":"41dde983-43b7-4c73-ae19-4955151dbf81","Type":"ContainerStarted","Data":"32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72"} Nov 01 08:45:12.674357 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.674258 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" event={"ID":"41dde983-43b7-4c73-ae19-4955151dbf81","Type":"ContainerStarted","Data":"0d7dffcd19018429806f7026dc200f4025c72f20a8e5f87e01ebdc433fecb348"} Nov 01 08:45:12.718067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718029 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv"] Nov 01 08:45:12.718523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718509 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" Nov 01 08:45:12.718565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718523 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36328be-0697-48b1-9ee3-9fc6970e9a8d" containerName="istio-proxy" Nov 01 08:45:12.718565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718537 5710 cpu_manager.go:414] "containerMap: removing container" podUID="fa78d778-0554-4cb5-9a10-261499cb73f6" containerName="revoke-keys" Nov 01 08:45:12.718565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718542 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa78d778-0554-4cb5-9a10-261499cb73f6" containerName="revoke-keys" Nov 01 08:45:12.718565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718550 5710 cpu_manager.go:414] "containerMap: removing container" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" Nov 01 08:45:12.718565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718557 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" Nov 01 08:45:12.718709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718618 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="244cff1b-680f-4c0b-a976-4e2127497800" containerName="istio-proxy" Nov 01 08:45:12.718709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.718627 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa78d778-0554-4cb5-9a10-261499cb73f6" containerName="revoke-keys" Nov 01 08:45:12.721999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.721978 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:12.789746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.784578 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-343207-dockercfg-6zs2j\"" Nov 01 08:45:12.789746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.784665 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-343207\"" Nov 01 08:45:12.789746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.787576 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv"] Nov 01 08:45:12.831601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.831536 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podStartSLOduration=1.8315168050000001 podStartE2EDuration="1.831516805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:12.78807303 +0000 UTC m=+735.876409725" watchObservedRunningTime="2026-11-01 08:45:12.831516805 +0000 UTC m=+735.919853478" Nov 01 08:45:12.856519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.856474 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:12.856746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.856586 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5j5\" (UniqueName: \"kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5\") pod \"payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:12.884136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.884100 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44"] Nov 01 08:45:12.888768 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.888708 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:12.942649 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.942556 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44"] Nov 01 08:45:12.957561 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.957523 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5j5\" (UniqueName: \"kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5\") pod \"payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:12.957797 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.957761 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:12.958472 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:12.958453 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:13.015359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.015318 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5j5\" (UniqueName: \"kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5\") pod \"payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:13.037019 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.036977 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:13.059121 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.059086 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfs6b\" (UniqueName: \"kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b\") pod \"payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.059322 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.059297 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.160710 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.160663 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.160928 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.160834 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfs6b\" (UniqueName: \"kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b\") pod \"payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.161396 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.161373 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.218748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.218641 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfs6b\" (UniqueName: \"kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b\") pod \"payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.413487 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.413446 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv"] Nov 01 08:45:13.421204 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:13.421170 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1131cda_c18a_4e17_a97a_771c88b65c9f.slice/crio-d9297f3b54c21e48d2abf43f5904052106847e6d41ce9f724ea0424bdc7f18b6 WatchSource:0}: Error finding container d9297f3b54c21e48d2abf43f5904052106847e6d41ce9f724ea0424bdc7f18b6: Status 404 returned error can't find the container with id d9297f3b54c21e48d2abf43f5904052106847e6d41ce9f724ea0424bdc7f18b6 Nov 01 08:45:13.517375 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.517342 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:13.573446 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.573420 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:13.584904 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.584876 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:13.682021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.681984 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" event={"ID":"e1131cda-c18a-4e17-a97a-771c88b65c9f","Type":"ContainerStarted","Data":"8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343"} Nov 01 08:45:13.682419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.682043 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" event={"ID":"e1131cda-c18a-4e17-a97a-771c88b65c9f","Type":"ContainerStarted","Data":"d9297f3b54c21e48d2abf43f5904052106847e6d41ce9f724ea0424bdc7f18b6"} Nov 01 08:45:13.682419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.682144 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:13.682419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.682190 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:13.683235 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.683211 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:13.807134 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.807020 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" podStartSLOduration=1.807005183 podStartE2EDuration="1.807005183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:13.805894963 +0000 UTC m=+736.894231636" watchObservedRunningTime="2026-11-01 08:45:13.807005183 +0000 UTC m=+736.895341855" Nov 01 08:45:13.890593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:13.890556 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44"] Nov 01 08:45:13.898158 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:13.898123 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e1360d_f860_472d_a024_dae5df31ed42.slice/crio-952cff860016c3de41efbc74ef1a124af6cfc994af64b88cfbf365637f66774c WatchSource:0}: Error finding container 952cff860016c3de41efbc74ef1a124af6cfc994af64b88cfbf365637f66774c: Status 404 returned error can't find the container with id 952cff860016c3de41efbc74ef1a124af6cfc994af64b88cfbf365637f66774c Nov 01 08:45:14.689410 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:14.689372 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" event={"ID":"93e1360d-f860-472d-a024-dae5df31ed42","Type":"ContainerStarted","Data":"66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa"} Nov 01 08:45:14.689937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:14.689436 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" event={"ID":"93e1360d-f860-472d-a024-dae5df31ed42","Type":"ContainerStarted","Data":"952cff860016c3de41efbc74ef1a124af6cfc994af64b88cfbf365637f66774c"} Nov 01 08:45:14.799359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:14.799301 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" podStartSLOduration=2.799284867 podStartE2EDuration="2.799284867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:14.797401654 +0000 UTC m=+737.885738350" watchObservedRunningTime="2026-11-01 08:45:14.799284867 +0000 UTC m=+737.887621539" Nov 01 08:45:15.477217 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:15.477182 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:45:15.696217 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:15.696183 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:17.262858 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.262826 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm"] Nov 01 08:45:17.267100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.267072 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.320823 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.320791 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-751a73\"" Nov 01 08:45:17.370958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.370924 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-751a73-dockercfg-bl77r\"" Nov 01 08:45:17.373057 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.373020 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm"] Nov 01 08:45:17.405839 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.405797 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.406030 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.406012 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26bf\" (UniqueName: \"kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf\") pod \"payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.427748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.427685 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk"] Nov 01 08:45:17.431600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.431579 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.492616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.492583 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk"] Nov 01 08:45:17.507761 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.507703 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26bf\" (UniqueName: \"kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf\") pod \"payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.507954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.507796 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.508707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.508681 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.566189 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.566110 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26bf\" (UniqueName: \"kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf\") pod \"payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.580260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.580225 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:17.609015 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.608981 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994dl\" (UniqueName: \"kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl\") pod \"payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.609171 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.609124 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.710607 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.710570 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.710800 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.710703 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994dl\" (UniqueName: \"kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl\") pod \"payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.711780 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.711754 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.768165 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.768134 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-994dl\" (UniqueName: \"kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl\") pod \"payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:17.955043 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:17.955007 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm"] Nov 01 08:45:17.963454 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:17.963423 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d357506_a8cc_4c6d_bdd5_1bc71f53045c.slice/crio-ce57dc86c5d903d25f51e02ac7b887ce8b0271f18f129f0e8844ea7c80cac62d WatchSource:0}: Error finding container ce57dc86c5d903d25f51e02ac7b887ce8b0271f18f129f0e8844ea7c80cac62d: Status 404 returned error can't find the container with id ce57dc86c5d903d25f51e02ac7b887ce8b0271f18f129f0e8844ea7c80cac62d Nov 01 08:45:18.047378 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.047351 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:18.422944 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.422907 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk"] Nov 01 08:45:18.430451 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:18.430418 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a8542e_ffb0_4dc5_b770_85d3205df3de.slice/crio-18284399245ba13c7e758e7d693791c0ab696dac9dab26447a05c33edd2875be WatchSource:0}: Error finding container 18284399245ba13c7e758e7d693791c0ab696dac9dab26447a05c33edd2875be: Status 404 returned error can't find the container with id 18284399245ba13c7e758e7d693791c0ab696dac9dab26447a05c33edd2875be Nov 01 08:45:18.714483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.714383 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" event={"ID":"0d357506-a8cc-4c6d-bdd5-1bc71f53045c","Type":"ContainerStarted","Data":"b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2"} Nov 01 08:45:18.714483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.714449 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" event={"ID":"0d357506-a8cc-4c6d-bdd5-1bc71f53045c","Type":"ContainerStarted","Data":"ce57dc86c5d903d25f51e02ac7b887ce8b0271f18f129f0e8844ea7c80cac62d"} Nov 01 08:45:18.714735 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.714571 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:18.716617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.716584 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" event={"ID":"e1a8542e-ffb0-4dc5-b770-85d3205df3de","Type":"ContainerStarted","Data":"ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349"} Nov 01 08:45:18.716821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.716634 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" event={"ID":"e1a8542e-ffb0-4dc5-b770-85d3205df3de","Type":"ContainerStarted","Data":"18284399245ba13c7e758e7d693791c0ab696dac9dab26447a05c33edd2875be"} Nov 01 08:45:18.716821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.716733 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:18.823226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:18.823158 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" podStartSLOduration=1.823142933 podStartE2EDuration="1.823142933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:18.822596197 +0000 UTC m=+741.910932881" watchObservedRunningTime="2026-11-01 08:45:18.823142933 +0000 UTC m=+741.911479605" Nov 01 08:45:21.051347 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:21.051313 5710 scope.go:122] "RemoveContainer" containerID="6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3" Nov 01 08:45:21.742553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:21.742460 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerStarted","Data":"83a25fbdff53c18e01ad332ce014e263a5e847993578f057f1c994c505607c2f"} Nov 01 08:45:21.852462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:21.852409 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" podStartSLOduration=4.852393205 podStartE2EDuration="4.852393205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:18.944997259 +0000 UTC m=+742.033333931" watchObservedRunningTime="2026-11-01 08:45:21.852393205 +0000 UTC m=+744.940729957" Nov 01 08:45:21.942790 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:21.942755 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85"] Nov 01 08:45:21.946799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:21.946774 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:21.995512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:21.995445 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-343207-serving-cert\"" Nov 01 08:45:22.001176 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.001142 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85"] Nov 01 08:45:22.060424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.060393 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.060876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.060438 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.060876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.060463 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.060876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.060660 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfm4j\" (UniqueName: \"kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.161417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.161385 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfm4j\" (UniqueName: \"kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.161594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.161480 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.161594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.161515 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.161594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.161536 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.162179 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.162153 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.164360 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.164329 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.164501 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.164385 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.220078 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.220049 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfm4j\" (UniqueName: \"kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j\") pod \"maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.259929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.259829 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:22.639876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.639803 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85"] Nov 01 08:45:22.648345 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:22.648303 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf82bf01_0af4_4bdd_9f81_941504980db7.slice/crio-4955c6731e18984a57e9d948126ca4e7b99275d1dc21b458008f580e5f6c448c WatchSource:0}: Error finding container 4955c6731e18984a57e9d948126ca4e7b99275d1dc21b458008f580e5f6c448c: Status 404 returned error can't find the container with id 4955c6731e18984a57e9d948126ca4e7b99275d1dc21b458008f580e5f6c448c Nov 01 08:45:22.750281 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.750237 5710 generic.go:356] "Generic (PLEG): container finished" podID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerID="83a25fbdff53c18e01ad332ce014e263a5e847993578f057f1c994c505607c2f" exitCode=60 Nov 01 08:45:22.750552 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.750313 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerDied","Data":"83a25fbdff53c18e01ad332ce014e263a5e847993578f057f1c994c505607c2f"} Nov 01 08:45:22.750552 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.750375 5710 scope.go:122] "RemoveContainer" containerID="6108ffba2c6ae2bde0676afa0bd61b0c499e879a3050c8b6db8818dcf7fd56d3" Nov 01 08:45:22.750839 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.750800 5710 scope.go:122] "RemoveContainer" containerID="83a25fbdff53c18e01ad332ce014e263a5e847993578f057f1c994c505607c2f" Nov 01 08:45:22.751142 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:22.751120 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h_odh-ai-gateway-infra(67fdd3b7-cdba-43e6-9c12-3e6b241571b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" Nov 01 08:45:22.752098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.752078 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" event={"ID":"af82bf01-0af4-4bdd-9f81-941504980db7","Type":"ContainerStarted","Data":"4955c6731e18984a57e9d948126ca4e7b99275d1dc21b458008f580e5f6c448c"} Nov 01 08:45:22.879001 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:22.878969 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h"] Nov 01 08:45:23.759421 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:23.759383 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" event={"ID":"af82bf01-0af4-4bdd-9f81-941504980db7","Type":"ContainerStarted","Data":"20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda"} Nov 01 08:45:23.759888 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:23.759455 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:23.911914 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:23.911883 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:23.983283 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:23.983209 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" podStartSLOduration=2.107129603 podStartE2EDuration="2.983185362s" totalImagesPullingTime="876.055759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:23.872499927 +0000 UTC m=+746.960836598" watchObservedRunningTime="2026-11-01 08:45:23.983185362 +0000 UTC m=+747.071522034" Nov 01 08:45:24.083530 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.083429 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm\" (UniqueName: \"kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm\") pod \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " Nov 01 08:45:24.083699 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.083558 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca\") pod \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\" (UID: \"67fdd3b7-cdba-43e6-9c12-3e6b241571b4\") " Nov 01 08:45:24.083919 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.083895 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca" pod "67fdd3b7-cdba-43e6-9c12-3e6b241571b4" (UID: "67fdd3b7-cdba-43e6-9c12-3e6b241571b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:24.086003 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.085975 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm" pod "67fdd3b7-cdba-43e6-9c12-3e6b241571b4" (UID: "67fdd3b7-cdba-43e6-9c12-3e6b241571b4"). InnerVolumeSpecName "kube-api-access-6cbrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:24.184831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.184797 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbrm\" (UniqueName: \"kubernetes.io/projected/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-kube-api-access-6cbrm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:24.184831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.184831 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67fdd3b7-cdba-43e6-9c12-3e6b241571b4-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:24.693386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.693357 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 08:45:24.769410 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.769379 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" Nov 01 08:45:24.769882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.769419 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h" event={"ID":"67fdd3b7-cdba-43e6-9c12-3e6b241571b4","Type":"ContainerDied","Data":"44ee943960753e3433436dbe7733bf1b6a0308733697788a606c1bcc18d1088c"} Nov 01 08:45:24.769882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.769484 5710 scope.go:122] "RemoveContainer" containerID="83a25fbdff53c18e01ad332ce014e263a5e847993578f057f1c994c505607c2f" Nov 01 08:45:24.906185 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.906148 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h"] Nov 01 08:45:24.912510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:24.912455 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-0f765cd6-29892045-p6h5h"] Nov 01 08:45:25.057785 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:25.057692 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" path="/var/lib/kubelet/pods/67fdd3b7-cdba-43e6-9c12-3e6b241571b4/volumes" Nov 01 08:45:26.272954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:26.272911 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:45:26.704647 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:26.704619 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 08:45:29.725565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:29.725537 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 08:45:29.726081 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:29.725603 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 08:45:29.775606 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:29.775576 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 08:45:41.791343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:41.791301 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4"] Nov 01 08:45:41.791929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:41.791544 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" podUID="7e668c98-ba64-4454-ab7e-c10fbc65517d" containerName="maas-api" containerID="cri-o://fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39" gracePeriod=30 Nov 01 08:45:42.067969 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.067940 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:45:42.193095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.192098 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca\") pod \"7e668c98-ba64-4454-ab7e-c10fbc65517d\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " Nov 01 08:45:42.193095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.192197 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs\") pod \"7e668c98-ba64-4454-ab7e-c10fbc65517d\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " Nov 01 08:45:42.193095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.192273 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\") pod \"7e668c98-ba64-4454-ab7e-c10fbc65517d\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " Nov 01 08:45:42.193095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.192325 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr\" (UniqueName: \"kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr\") pod \"7e668c98-ba64-4454-ab7e-c10fbc65517d\" (UID: \"7e668c98-ba64-4454-ab7e-c10fbc65517d\") " Nov 01 08:45:42.195736 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.195682 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca" pod "7e668c98-ba64-4454-ab7e-c10fbc65517d" (UID: "7e668c98-ba64-4454-ab7e-c10fbc65517d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:42.204174 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.204111 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs" pod "7e668c98-ba64-4454-ab7e-c10fbc65517d" (UID: "7e668c98-ba64-4454-ab7e-c10fbc65517d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:42.204835 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.204796 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr" pod "7e668c98-ba64-4454-ab7e-c10fbc65517d" (UID: "7e668c98-ba64-4454-ab7e-c10fbc65517d"). InnerVolumeSpecName "kube-api-access-gqjjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:42.206630 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.206601 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls" pod "7e668c98-ba64-4454-ab7e-c10fbc65517d" (UID: "7e668c98-ba64-4454-ab7e-c10fbc65517d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:45:42.299993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.293785 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:42.299993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.293826 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjjr\" (UniqueName: \"kubernetes.io/projected/7e668c98-ba64-4454-ab7e-c10fbc65517d-kube-api-access-gqjjr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:42.299993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.293843 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e668c98-ba64-4454-ab7e-c10fbc65517d-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:42.299993 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.293856 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e668c98-ba64-4454-ab7e-c10fbc65517d-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:42.877195 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.877152 5710 generic.go:356] "Generic (PLEG): container finished" podID="7e668c98-ba64-4454-ab7e-c10fbc65517d" containerID="fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39" exitCode=0 Nov 01 08:45:42.877669 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.877201 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" Nov 01 08:45:42.877669 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.877228 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" event={"ID":"7e668c98-ba64-4454-ab7e-c10fbc65517d","Type":"ContainerDied","Data":"fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39"} Nov 01 08:45:42.877669 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.877282 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4" event={"ID":"7e668c98-ba64-4454-ab7e-c10fbc65517d","Type":"ContainerDied","Data":"07ada6e7d92ca48b5f0b8493d9d54d90dc0b29a200e5eb476dd5b89b8be07f69"} Nov 01 08:45:42.877669 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.877306 5710 scope.go:122] "RemoveContainer" containerID="fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39" Nov 01 08:45:42.887667 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.887645 5710 scope.go:122] "RemoveContainer" containerID="fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39" Nov 01 08:45:42.888079 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:42.888055 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39\": container with ID starting with fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39 not found: ID does not exist" containerID="fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39" Nov 01 08:45:42.888135 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:42.888092 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39"} err="failed to get container status \"fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39\": rpc error: code = NotFound desc = could not find container \"fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39\": container with ID starting with fdb6bc40e1e3b4071f8aeb7657ce5470526ca67ab247a8caf092f2654ed18f39 not found: ID does not exist" Nov 01 08:45:43.031874 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:43.031840 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4"] Nov 01 08:45:43.078074 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:43.078029 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f765cd6-684fd9889c-rksk4"] Nov 01 08:45:43.685160 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:43.685125 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:45:43.962529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:43.962418 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6"] Nov 01 08:45:44.011148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:44.011079 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f765cd6-kdgz6"] Nov 01 08:45:44.018930 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:44.018885 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs"] Nov 01 08:45:44.019148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:44.019116 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" containerID="cri-o://3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85" gracePeriod=30 Nov 01 08:45:45.057379 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.057345 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e668c98-ba64-4454-ab7e-c10fbc65517d" path="/var/lib/kubelet/pods/7e668c98-ba64-4454-ab7e-c10fbc65517d/volumes" Nov 01 08:45:45.057826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.057777 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa78d778-0554-4cb5-9a10-261499cb73f6" path="/var/lib/kubelet/pods/fa78d778-0554-4cb5-9a10-261499cb73f6/volumes" Nov 01 08:45:45.408697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.408660 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq"] Nov 01 08:45:45.409707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409668 5710 cpu_manager.go:414] "containerMap: removing container" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409706 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409775 5710 cpu_manager.go:414] "containerMap: removing container" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409784 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409822 5710 cpu_manager.go:414] "containerMap: removing container" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409830 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409847 5710 cpu_manager.go:414] "containerMap: removing container" podUID="7e668c98-ba64-4454-ab7e-c10fbc65517d" containerName="maas-api" Nov 01 08:45:45.409893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409855 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e668c98-ba64-4454-ab7e-c10fbc65517d" containerName="maas-api" Nov 01 08:45:45.410136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409962 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e668c98-ba64-4454-ab7e-c10fbc65517d" containerName="maas-api" Nov 01 08:45:45.410136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409979 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.410136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.409990 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:45.415907 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.415871 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.463513 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.463481 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq"] Nov 01 08:45:45.491625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.491593 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfdd13-openshift-default-dockercfg-jf87f\"" Nov 01 08:45:45.512488 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:45.512443 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:45:45.527518 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.527475 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.527518 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.527524 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96kg\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.527785 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.527670 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.527785 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.527761 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.527853 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.527803 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.527951 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.527932 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.528037 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.528022 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.528106 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.528069 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.528140 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.528127 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.629773 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629711 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.629773 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629777 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629824 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629855 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96kg\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629912 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629942 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.629971 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630274 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.630050 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630274 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.630124 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630603 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.630575 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630911 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.630766 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.630911 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.630846 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.631077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.631030 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.631077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.631042 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.633034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.633008 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.633230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.633212 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.711076 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.710969 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96kg\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.715034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.712338 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:45.732268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:45.732226 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:46.200762 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:46.200076 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq"] Nov 01 08:45:46.209282 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:46.209251 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4eb97f_3177_47da_befe_2cde7b743657.slice/crio-85f79c9413a25d82fa243690714d43ba1e045a1bc13c746829f67cadab8bbba0 WatchSource:0}: Error finding container 85f79c9413a25d82fa243690714d43ba1e045a1bc13c746829f67cadab8bbba0: Status 404 returned error can't find the container with id 85f79c9413a25d82fa243690714d43ba1e045a1bc13c746829f67cadab8bbba0 Nov 01 08:45:46.212169 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:46.212135 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:46.212281 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:46.212221 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:46.212281 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:46.212265 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:46.926314 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:46.926276 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" event={"ID":"0e4eb97f-3177-47da-befe-2cde7b743657","Type":"ContainerStarted","Data":"aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0"} Nov 01 08:45:46.926560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:46.926339 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" event={"ID":"0e4eb97f-3177-47da-befe-2cde7b743657","Type":"ContainerStarted","Data":"85f79c9413a25d82fa243690714d43ba1e045a1bc13c746829f67cadab8bbba0"} Nov 01 08:45:47.040474 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:47.040426 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podStartSLOduration=2.040410753 podStartE2EDuration="2.040410753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:47.03928155 +0000 UTC m=+770.127618222" watchObservedRunningTime="2026-11-01 08:45:47.040410753 +0000 UTC m=+770.128747424" Nov 01 08:45:47.732901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:47.732859 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:47.739037 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:47.738997 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:47.932044 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:47.932019 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:47.933192 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:47.933173 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:45:49.335230 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.335201 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:45:49.368428 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368390 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368452 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368482 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368527 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368557 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368579 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368939 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368634 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368939 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368669 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.368939 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368713 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket\") pod \"18d23143-aea5-4424-800e-9e94acd29808\" (UID: \"18d23143-aea5-4424-800e-9e94acd29808\") " Nov 01 08:45:49.369087 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.368992 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:45:49.369148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.369084 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:49.369148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.369028 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:49.369250 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.369229 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:49.369312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.369274 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:49.371268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.371231 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:45:49.371565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.371549 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "kube-api-access-l574w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:49.371565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.371549 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:45:49.372089 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.372070 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo" pod "18d23143-aea5-4424-800e-9e94acd29808" (UID: "18d23143-aea5-4424-800e-9e94acd29808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 08:45:49.470285 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470238 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470285 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470276 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l574w\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-kube-api-access-l574w\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470285 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470290 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d23143-aea5-4424-800e-9e94acd29808-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470299 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470308 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470317 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470325 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d23143-aea5-4424-800e-9e94acd29808-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470335 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d23143-aea5-4424-800e-9e94acd29808-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.470547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.470343 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d23143-aea5-4424-800e-9e94acd29808-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:45:49.945860 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.945823 5710 generic.go:356] "Generic (PLEG): container finished" podID="18d23143-aea5-4424-800e-9e94acd29808" containerID="3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85" exitCode=0 Nov 01 08:45:49.946063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.945930 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" Nov 01 08:45:49.946121 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.945912 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" event={"ID":"18d23143-aea5-4424-800e-9e94acd29808","Type":"ContainerDied","Data":"3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85"} Nov 01 08:45:49.946182 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.946159 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs" event={"ID":"18d23143-aea5-4424-800e-9e94acd29808","Type":"ContainerDied","Data":"f4817827f5c2b30e29b80aedaa25760bd970b79d9610c2ce65a4b9c8662dcc19"} Nov 01 08:45:49.946232 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.946194 5710 scope.go:122] "RemoveContainer" containerID="3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85" Nov 01 08:45:49.957025 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.956961 5710 scope.go:122] "RemoveContainer" containerID="3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85" Nov 01 08:45:49.957421 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:45:49.957399 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85\": container with ID starting with 3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85 not found: ID does not exist" containerID="3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85" Nov 01 08:45:49.957484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:49.957431 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85"} err="failed to get container status \"3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85\": rpc error: code = NotFound desc = could not find container \"3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85\": container with ID starting with 3ae4e506ed990a3122240637047dda589fc062c49165e633460ef985d02a4c85 not found: ID does not exist" Nov 01 08:45:50.100737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:50.100684 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs"] Nov 01 08:45:50.146877 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:50.146843 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0f765cd6-openshift-default-6ccb956fd5-qpgvs"] Nov 01 08:45:51.058912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:51.058876 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d23143-aea5-4424-800e-9e94acd29808" path="/var/lib/kubelet/pods/18d23143-aea5-4424-800e-9e94acd29808/volumes" Nov 01 08:45:53.972551 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:53.972511 5710 generic.go:356] "Generic (PLEG): container finished" podID="41dde983-43b7-4c73-ae19-4955151dbf81" containerID="32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72" exitCode=0 Nov 01 08:45:53.973061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:53.972581 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" event={"ID":"41dde983-43b7-4c73-ae19-4955151dbf81","Type":"ContainerDied","Data":"32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72"} Nov 01 08:45:53.973061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:53.973010 5710 scope.go:122] "RemoveContainer" containerID="32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72" Nov 01 08:45:53.974325 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:53.974298 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:53.974381 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:53.974369 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:53.974424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:53.974400 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:45:54.980449 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:54.980412 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" event={"ID":"41dde983-43b7-4c73-ae19-4955151dbf81","Type":"ContainerStarted","Data":"5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa"} Nov 01 08:45:55.573510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:55.573462 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:55.575092 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:55.575066 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 08:45:55.575256 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:55.575118 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 08:45:56.339709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.339671 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv"] Nov 01 08:45:56.340195 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.340180 5710 cpu_manager.go:414] "containerMap: removing container" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" Nov 01 08:45:56.340241 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.340196 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" Nov 01 08:45:56.340313 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.340303 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fdd3b7-cdba-43e6-9c12-3e6b241571b4" containerName="cleanup" Nov 01 08:45:56.340348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.340314 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d23143-aea5-4424-800e-9e94acd29808" containerName="istio-proxy" Nov 01 08:45:56.343827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.343788 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.399889 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.399849 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dfdd13\"" Nov 01 08:45:56.411965 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.411927 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv"] Nov 01 08:45:56.448309 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.448269 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dfdd13-dockercfg-6zpl8\"" Nov 01 08:45:56.456091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.456037 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.456444 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.456423 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt7d\" (UniqueName: \"kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d\") pod \"payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.557992 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.557951 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.557992 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.558002 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt7d\" (UniqueName: \"kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d\") pod \"payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.558930 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.558898 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.573403 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.573346 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 08:45:56.573609 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.573417 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 08:45:56.616174 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.616139 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt7d\" (UniqueName: \"kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d\") pod \"payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:56.658278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:56.658238 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:57.080097 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:57.080021 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv"] Nov 01 08:45:57.110212 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:45:57.110175 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81fbead_001d_408c_946b_0bff9281b071.slice/crio-6f47eb1256567ec4c642013ef5068219ad25ffc1432ee05a359b1478a8c141b5 WatchSource:0}: Error finding container 6f47eb1256567ec4c642013ef5068219ad25ffc1432ee05a359b1478a8c141b5: Status 404 returned error can't find the container with id 6f47eb1256567ec4c642013ef5068219ad25ffc1432ee05a359b1478a8c141b5 Nov 01 08:45:57.573443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:57.573405 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 08:45:57.573909 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:57.573460 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 08:45:58.026487 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.026447 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" event={"ID":"a81fbead-001d-408c-946b-0bff9281b071","Type":"ContainerStarted","Data":"4ede7eaa58dc1569cb14752cfdd04191193fb75eac6a0b0286a77084580fc7b7"} Nov 01 08:45:58.026689 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.026532 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" event={"ID":"a81fbead-001d-408c-946b-0bff9281b071","Type":"ContainerStarted","Data":"6f47eb1256567ec4c642013ef5068219ad25ffc1432ee05a359b1478a8c141b5"} Nov 01 08:45:58.026689 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.026554 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:45:58.138976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.138908 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" podStartSLOduration=2.138887797 podStartE2EDuration="2.138887797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:45:58.136374136 +0000 UTC m=+781.224710809" watchObservedRunningTime="2026-11-01 08:45:58.138887797 +0000 UTC m=+781.227224470" Nov 01 08:45:58.447763 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.447708 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:45:58.573577 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.573536 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 08:45:58.574217 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.574183 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 08:45:58.701819 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:58.699183 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:45:59.574154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:59.574113 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 08:45:59.574589 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:45:59.574177 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 08:46:00.578540 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:00.578503 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:46:00.579783 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:00.579760 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 08:46:05.941159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:05.941117 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:46:09.043954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:09.043917 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 08:46:15.478414 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:46:15.478369 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:46:18.918565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:18.918527 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg"] Nov 01 08:46:18.922397 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:18.922374 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:18.971049 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:18.971016 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-dfdd13-serving-cert\"" Nov 01 08:46:19.019350 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.019315 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg"] Nov 01 08:46:19.030512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.030473 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.030759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.030635 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.030759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.030674 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9ww\" (UniqueName: \"kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.030886 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.030808 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.131477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.131440 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.131477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.131484 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9ww\" (UniqueName: \"kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.131808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.131505 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.131808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.131566 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.131808 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:46:19.131610 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-serving-cert: secret "maas-api-e2e-worker-w1-dfdd13-serving-cert" not found Nov 01 08:46:19.131808 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:46:19.131697 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls podName:3b162c79-4e90-4a17-a3ca-d0996729e047 nodeName:}" failed. No retries permitted until 2026-11-01 08:46:19.631674198 +0000 UTC m=+802.720010863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls") pod "maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" (UID: "3b162c79-4e90-4a17-a3ca-d0996729e047") : secret "maas-api-e2e-worker-w1-dfdd13-serving-cert" not found Nov 01 08:46:19.132344 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.132316 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.134737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.134703 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.189705 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.189627 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9ww\" (UniqueName: \"kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.638284 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.638236 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.641186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.641152 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:19.836170 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:19.836125 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:20.221101 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:20.220998 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg"] Nov 01 08:46:21.159184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:21.159140 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" event={"ID":"3b162c79-4e90-4a17-a3ca-d0996729e047","Type":"ContainerStarted","Data":"7f4c8c70cdc6ec8284fdd9a70a7f75e5b8f2a094badf69f5d3f313a4a4c0bf18"} Nov 01 08:46:22.166298 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:22.166259 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" event={"ID":"3b162c79-4e90-4a17-a3ca-d0996729e047","Type":"ContainerStarted","Data":"ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8"} Nov 01 08:46:22.166734 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:22.166393 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:22.277519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:22.277456 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" podStartSLOduration=3.419943935 podStartE2EDuration="4.277434993s" totalImagesPullingTime="857.491058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:46:22.276747094 +0000 UTC m=+805.365083766" watchObservedRunningTime="2026-11-01 08:46:22.277434993 +0000 UTC m=+805.365771666" Nov 01 08:46:23.647198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:23.647155 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:46:28.177001 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:28.176974 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 08:46:39.708831 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:39.708795 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:46:40.264574 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:40.264544 5710 generic.go:356] "Generic (PLEG): container finished" podID="0e4eb97f-3177-47da-befe-2cde7b743657" containerID="aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0" exitCode=0 Nov 01 08:46:40.264772 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:40.264592 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" event={"ID":"0e4eb97f-3177-47da-befe-2cde7b743657","Type":"ContainerDied","Data":"aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0"} Nov 01 08:46:40.264982 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:40.264969 5710 scope.go:122] "RemoveContainer" containerID="aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0" Nov 01 08:46:40.266248 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:40.266222 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:46:40.266323 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:40.266291 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:46:40.266362 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:40.266321 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:46:41.271706 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:41.271668 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" event={"ID":"0e4eb97f-3177-47da-befe-2cde7b743657","Type":"ContainerStarted","Data":"20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32"} Nov 01 08:46:41.732644 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:41.732601 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:46:41.732884 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:41.732835 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 08:46:41.732934 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:41.732888 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 08:46:42.733129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:42.733093 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 08:46:42.733550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:42.733157 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 08:46:43.733455 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:43.733419 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 08:46:43.733908 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:43.733480 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 08:46:44.732789 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:44.732754 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 08:46:44.733012 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:44.732821 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 08:46:45.485008 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:46:45.484967 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:46:45.733405 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:45.733372 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 08:46:45.733610 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:45.733426 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 08:46:46.737529 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:46.737497 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:46:46.737987 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:46.737754 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:46:46.738697 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:46.738677 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 08:46:50.589756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:46:50.589706 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:47:04.268056 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:04.268009 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:47:15.478544 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:47:15.478496 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:47:27.678860 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:27.678816 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:47:29.916520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:29.916468 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn"] Nov 01 08:47:29.923670 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:29.923639 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:29.974098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:29.974063 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn"] Nov 01 08:47:30.017866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.017835 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"kube-root-ca.crt\"" Nov 01 08:47:30.017866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.017858 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"openshift-service-ca.crt\"" Nov 01 08:47:30.018083 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.017899 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"default-dockercfg-l5p8k\"" Nov 01 08:47:30.018811 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.018791 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"simulator-w1-dfdd13-kserve-self-signed-certs\"" Nov 01 08:47:30.034300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.034276 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.034399 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.034319 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw829\" (UniqueName: \"kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.034468 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.034421 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.034534 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.034518 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.034608 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.034580 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.034608 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.034600 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135250 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135313 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw829\" (UniqueName: \"kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135362 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135411 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135515 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135543 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135851 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135704 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.135963 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.135935 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.136046 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.136022 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.137952 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.137911 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.138260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.138242 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.196348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.196272 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw829\" (UniqueName: \"kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829\") pod \"simulator-w1-dfdd13-kserve-75459998d6-kfzpn\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.237277 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.237239 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:30.615416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:30.615380 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn"] Nov 01 08:47:30.623898 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:47:30.623859 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0805ce31_0624_45fe_a11d_31e49ec7b01b.slice/crio-4bffc67c8cac06555b87bb05faf164844ba556c86a2d86edbd9841a89c2c5acb WatchSource:0}: Error finding container 4bffc67c8cac06555b87bb05faf164844ba556c86a2d86edbd9841a89c2c5acb: Status 404 returned error can't find the container with id 4bffc67c8cac06555b87bb05faf164844ba556c86a2d86edbd9841a89c2c5acb Nov 01 08:47:31.577997 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:31.577957 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" event={"ID":"0805ce31-0624-45fe-a11d-31e49ec7b01b","Type":"ContainerStarted","Data":"f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a"} Nov 01 08:47:31.578467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:31.578010 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" event={"ID":"0805ce31-0624-45fe-a11d-31e49ec7b01b","Type":"ContainerStarted","Data":"4bffc67c8cac06555b87bb05faf164844ba556c86a2d86edbd9841a89c2c5acb"} Nov 01 08:47:31.689063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:31.689007 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podStartSLOduration=2.688991925 podStartE2EDuration="2.688991925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:31.687116028 +0000 UTC m=+874.775452701" watchObservedRunningTime="2026-11-01 08:47:31.688991925 +0000 UTC m=+874.777328597" Nov 01 08:47:32.237475 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:32.237436 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:37.239922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.239889 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z"] Nov 01 08:47:37.243987 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.243965 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.293011 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.292976 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-751a73-openshift-default-dockercfg-qq2cf\"" Nov 01 08:47:37.323316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323275 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323535 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323342 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323535 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323424 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323535 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323482 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323572 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323603 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27nx\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323678 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323744 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.323821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.323788 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.343460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.343425 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z"] Nov 01 08:47:37.424958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.424917 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.424975 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425012 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425055 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425093 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425114 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425175 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425209 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425236 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27nx\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425535 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425665 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425639 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425712 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425672 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.425966 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.425814 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.426113 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.426097 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.427542 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.427515 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.427664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.427648 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.484813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.484784 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.485004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.484973 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27nx\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.558971 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.558892 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:37.987858 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.987818 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z"] Nov 01 08:47:37.995376 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:47:37.995337 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8028b953_7c08_4b94_9832_9c979607609d.slice/crio-b759afcff1649b144b9db0bad19fa54f01b6c2fa5ae58b24aa7b4f8e44b0c46c WatchSource:0}: Error finding container b759afcff1649b144b9db0bad19fa54f01b6c2fa5ae58b24aa7b4f8e44b0c46c: Status 404 returned error can't find the container with id b759afcff1649b144b9db0bad19fa54f01b6c2fa5ae58b24aa7b4f8e44b0c46c Nov 01 08:47:37.997952 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.997917 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:47:37.998045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.997987 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:47:37.998045 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:37.998015 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:47:38.639109 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:38.639066 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" event={"ID":"8028b953-7c08-4b94-9832-9c979607609d","Type":"ContainerStarted","Data":"7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b"} Nov 01 08:47:38.639564 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:38.639128 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" event={"ID":"8028b953-7c08-4b94-9832-9c979607609d","Type":"ContainerStarted","Data":"b759afcff1649b144b9db0bad19fa54f01b6c2fa5ae58b24aa7b4f8e44b0c46c"} Nov 01 08:47:38.804549 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:38.804477 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podStartSLOduration=1.804456228 podStartE2EDuration="1.804456228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:38.803544666 +0000 UTC m=+881.891881353" watchObservedRunningTime="2026-11-01 08:47:38.804456228 +0000 UTC m=+881.892792902" Nov 01 08:47:39.559711 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:39.559674 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:39.561465 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:39.561438 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 08:47:39.561603 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:39.561492 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 08:47:40.237396 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:40.237352 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:40.250551 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:40.250510 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:40.560091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:40.560007 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 08:47:40.560091 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:40.560072 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 08:47:40.663277 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:40.663242 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:47:41.560291 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:41.560254 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 08:47:41.560812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:41.560334 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 08:47:42.559413 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:42.559377 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 08:47:42.559605 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:42.559449 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 08:47:43.560161 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:43.560121 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 08:47:43.560694 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:43.560183 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 08:47:44.559919 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:44.559881 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 08:47:44.560111 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:44.559952 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 08:47:45.477530 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:47:45.477487 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:47:45.565630 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:45.565602 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:45.566040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:45.565998 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:45.566935 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:45.566914 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 08:47:45.884918 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:45.884880 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:47:50.714143 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.714106 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l"] Nov 01 08:47:50.718731 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.718678 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.768416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.768382 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-343207-openshift-default-dockercfg-j8ktj\"" Nov 01 08:47:50.807941 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.807893 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwjq\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.807980 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808044 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808093 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808118 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808186 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808210 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808235 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.808397 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.808363 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.817233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.817200 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l"] Nov 01 08:47:50.909530 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909497 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909530 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909547 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909591 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909616 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909640 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909663 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909695 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.909826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909795 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.910135 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.909833 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwjq\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.910198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.910146 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.910316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.910292 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.910386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.910338 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.910442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.910422 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.911026 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.910990 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.912268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.912242 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.912373 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.912332 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.968024 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.967957 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwjq\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:50.968024 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:50.967984 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:51.034295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.034257 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:51.480905 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.480867 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l"] Nov 01 08:47:51.488813 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:47:51.488768 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode429971d_6ae7_4d59_9de4_81d00c4bf43b.slice/crio-378d914ca64eaf4c0c4f0b75ba6db9515c2b5e4b6bb76d9ddcf1229188043174 WatchSource:0}: Error finding container 378d914ca64eaf4c0c4f0b75ba6db9515c2b5e4b6bb76d9ddcf1229188043174: Status 404 returned error can't find the container with id 378d914ca64eaf4c0c4f0b75ba6db9515c2b5e4b6bb76d9ddcf1229188043174 Nov 01 08:47:51.491434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.491402 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:47:51.491513 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.491473 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:47:51.491513 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.491504 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:47:51.721027 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.720992 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" event={"ID":"e429971d-6ae7-4d59-9de4-81d00c4bf43b","Type":"ContainerStarted","Data":"6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609"} Nov 01 08:47:51.721536 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.721057 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" event={"ID":"e429971d-6ae7-4d59-9de4-81d00c4bf43b","Type":"ContainerStarted","Data":"378d914ca64eaf4c0c4f0b75ba6db9515c2b5e4b6bb76d9ddcf1229188043174"} Nov 01 08:47:51.836244 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:51.836135 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podStartSLOduration=1.836113117 podStartE2EDuration="1.836113117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:47:51.834272533 +0000 UTC m=+894.922609207" watchObservedRunningTime="2026-11-01 08:47:51.836113117 +0000 UTC m=+894.924449788" Nov 01 08:47:52.035258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:52.035213 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:52.036667 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:52.036635 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:52.036837 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:52.036686 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:53.035481 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:53.035443 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:53.035928 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:53.035505 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:54.035737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:54.035636 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:54.036152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:54.035773 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:55.035335 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:55.035277 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:55.035570 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:55.035351 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:56.035652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:56.035616 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:56.036106 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:56.035664 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:57.035078 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:57.035038 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:57.035394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:57.035363 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:57.230564 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:57.230531 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:47:57.240259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:57.240234 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:47:58.035086 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:58.035050 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 01 08:47:58.035326 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:58.035104 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 01 08:47:59.531605 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:59.531557 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:59.532608 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:59.532570 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:47:59.534148 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:47:59.534127 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 08:48:03.439992 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:03.439939 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:48:15.501918 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:48:15.501877 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:48:25.693361 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:25.693231 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:48:42.971067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:42.971027 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d"] Nov 01 08:48:42.975143 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:42.975121 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.072991 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.072942 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d"] Nov 01 08:48:43.092639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.092595 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.092639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.092644 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.092894 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.092823 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.092894 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.092886 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.092983 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.092969 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.093021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.093000 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.093061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.093037 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7mj\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.093188 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.093164 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.093267 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.093251 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194339 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194409 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194491 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194514 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194548 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194582 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194648 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194687 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194745 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7mj\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.194852 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.195104 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.195929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.195822 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.196800 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.196072 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.196800 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.196386 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.201349 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.201319 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.207704 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.207671 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.258180 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.258054 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7mj\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.259622 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.259557 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.294309 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.293947 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:43.747092 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.747056 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d"] Nov 01 08:48:43.803579 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.803536 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:48:43.803855 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.803830 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:48:43.804024 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:43.804004 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:48:44.069431 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:44.069343 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" event={"ID":"5b71a780-fc73-4bcf-979e-16bcd0ba5765","Type":"ContainerStarted","Data":"454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788"} Nov 01 08:48:44.070128 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:44.069921 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" event={"ID":"5b71a780-fc73-4bcf-979e-16bcd0ba5765","Type":"ContainerStarted","Data":"e11e2ffbf33dc75b1a361b12dd94908adca7128937e5d755250fe2e7f807ea06"} Nov 01 08:48:44.185748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:44.185653 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podStartSLOduration=2.185631765 podStartE2EDuration="2.185631765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:44.183113178 +0000 UTC m=+947.271449862" watchObservedRunningTime="2026-11-01 08:48:44.185631765 +0000 UTC m=+947.273968436" Nov 01 08:48:44.294404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:44.294362 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:44.296300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:44.296266 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:44.296488 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:44.296340 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:45.296318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:45.296278 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:45.297377 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:45.296950 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:45.478641 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:48:45.478602 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:48:46.306146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:46.306100 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:46.306705 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:46.306176 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:47.297221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:47.297175 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:47.297555 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:47.297519 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:48.297890 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:48.297849 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:48.298493 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:48.298459 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:49.296259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:49.296208 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:49.296500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:49.296280 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:50.295650 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:50.295612 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 08:48:50.296122 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:50.295682 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 08:48:51.299618 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:51.299585 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:51.300027 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:51.299988 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:51.300794 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:51.300766 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 08:48:55.517571 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.517525 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b"] Nov 01 08:48:55.522430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.522397 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.571607 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.571572 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-54e01b-openshift-default-dockercfg-gln2n\"" Nov 01 08:48:55.600189 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600138 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600271 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600331 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600417 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600450 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600523 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcsqk\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600574 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600629 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.600683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.600674 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.620980 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.620946 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b"] Nov 01 08:48:55.701700 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.701662 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.701700 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.701729 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.701971 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.701840 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.701971 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.701928 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.701971 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.701961 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702002 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcsqk\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702047 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702088 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702096 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702150 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702317 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702270 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702341 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702499 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.702864 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.702838 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.704274 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.704257 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.704539 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.704523 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.762231 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.762198 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.762367 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.762267 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcsqk\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:55.837278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:55.837196 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:56.288123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:56.288084 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b"] Nov 01 08:48:56.295479 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:48:56.295448 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e47564_b224_41c0_b438_6c00d3dec913.slice/crio-a819528a4428d19d4d493b7bb5cebca46a836812dd8b142bd64084fea663c23e WatchSource:0}: Error finding container a819528a4428d19d4d493b7bb5cebca46a836812dd8b142bd64084fea663c23e: Status 404 returned error can't find the container with id a819528a4428d19d4d493b7bb5cebca46a836812dd8b142bd64084fea663c23e Nov 01 08:48:56.297904 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:56.297874 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:48:56.297999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:56.297940 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:48:56.297999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:56.297974 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:48:57.178572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:57.178538 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" event={"ID":"a8e47564-b224-41c0-b438-6c00d3dec913","Type":"ContainerStarted","Data":"5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d"} Nov 01 08:48:57.179186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:57.178594 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" event={"ID":"a8e47564-b224-41c0-b438-6c00d3dec913","Type":"ContainerStarted","Data":"a819528a4428d19d4d493b7bb5cebca46a836812dd8b142bd64084fea663c23e"} Nov 01 08:48:57.338571 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:57.338511 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podStartSLOduration=2.338495176 podStartE2EDuration="2.338495176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:48:57.338184443 +0000 UTC m=+960.426521125" watchObservedRunningTime="2026-11-01 08:48:57.338495176 +0000 UTC m=+960.426831848" Nov 01 08:48:57.837905 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:57.837862 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:48:57.839808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:57.839780 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 08:48:57.839966 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:57.839846 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 08:48:58.525146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:58.525100 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:48:58.838477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:58.838318 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 08:48:58.839110 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:58.839064 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 08:48:59.846701 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:59.844180 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 08:48:59.846701 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:48:59.844247 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 08:49:00.838683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:00.838638 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 08:49:00.838879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:00.838712 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 08:49:01.838530 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:01.838491 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 08:49:01.838977 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:01.838561 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 08:49:02.837690 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:02.837651 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 08:49:02.837952 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:02.837703 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 08:49:03.842648 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:03.842613 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:49:03.843119 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:03.842947 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:49:03.843804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:03.843786 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 08:49:15.478589 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:49:15.478553 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:49:20.498154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:20.498119 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:49:43.459560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.459478 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc"] Nov 01 08:49:43.463583 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.463561 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.561358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.561327 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 08:49:43.561358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.561337 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 08:49:43.561358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.561355 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 08:49:43.561358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.561326 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f77f55536278\"" Nov 01 08:49:43.561358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.561327 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 08:49:43.561740 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.561411 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d7bf9\"" Nov 01 08:49:43.563678 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.563657 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc"] Nov 01 08:49:43.578739 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.578693 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.578925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.578810 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvz9\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.578925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.578885 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.579042 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.579015 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.679685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.679638 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.679685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.679695 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvz9\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.680040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.679877 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.680040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.679990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.680515 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.680477 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.680681 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.680593 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.682914 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.682888 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.737595 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.737519 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvz9\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9\") pod \"payload-processing-external-model-74bd68c8cc-8xwxc\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:43.777263 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:43.777224 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:44.157074 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:44.157031 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc"] Nov 01 08:49:44.164997 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:49:44.164966 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3db70da_51ec_4e7c_b12b_bff305405337.slice/crio-ed52d244407f77115c56f0d8e4b1c18bb18eed7c9ea7d44c82e48133d8d84d63 WatchSource:0}: Error finding container ed52d244407f77115c56f0d8e4b1c18bb18eed7c9ea7d44c82e48133d8d84d63: Status 404 returned error can't find the container with id ed52d244407f77115c56f0d8e4b1c18bb18eed7c9ea7d44c82e48133d8d84d63 Nov 01 08:49:44.550429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:44.550329 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" event={"ID":"e3db70da-51ec-4e7c-b12b-bff305405337","Type":"ContainerStarted","Data":"6d37421240030f24acc4ddf83eba56e32785e67aab0c05f1723ffbd4a29b10a7"} Nov 01 08:49:44.550429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:44.550386 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" event={"ID":"e3db70da-51ec-4e7c-b12b-bff305405337","Type":"ContainerStarted","Data":"ed52d244407f77115c56f0d8e4b1c18bb18eed7c9ea7d44c82e48133d8d84d63"} Nov 01 08:49:44.550429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:44.550428 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:49:44.660737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:44.660655 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" podStartSLOduration=1.6606383120000001 podStartE2EDuration="1.660638312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:44.658587305 +0000 UTC m=+1007.746923998" watchObservedRunningTime="2026-11-01 08:49:44.660638312 +0000 UTC m=+1007.748974983" Nov 01 08:49:45.477885 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:49:45.477850 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:49:49.014823 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:49.014789 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:49:53.953159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:53.953126 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4"] Nov 01 08:49:53.957523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:53.957498 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021439 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021389 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021515 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021563 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpgx7\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021585 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021686 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021759 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021806 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021834 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.021931 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.021918 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.055242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.055181 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4"] Nov 01 08:49:54.122755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.122702 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.122755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.122761 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123015 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.122782 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123015 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.122820 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123161 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123049 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123161 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123139 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123186 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpgx7\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123239 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123249 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123345 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123440 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.123704 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.123685 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.124054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.124035 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.124163 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.124140 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.125948 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.125921 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.126149 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.126132 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.185318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.185285 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.185485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.185406 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpgx7\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7\") pod \"maas-default-gateway-openshift-default-746ff8f49f-292m4\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.271394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.271311 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:54.701443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.701398 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4"] Nov 01 08:49:54.708818 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:49:54.708784 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a6db99_8c3d_4882_bf3f_35a6269f9a71.slice/crio-0c4f9700e52a17e3c0898fe6d878d4e6147f89ebbb15c34c84e93c92084f5723 WatchSource:0}: Error finding container 0c4f9700e52a17e3c0898fe6d878d4e6147f89ebbb15c34c84e93c92084f5723: Status 404 returned error can't find the container with id 0c4f9700e52a17e3c0898fe6d878d4e6147f89ebbb15c34c84e93c92084f5723 Nov 01 08:49:54.711145 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.711112 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:49:54.711288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.711191 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:49:54.711288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:54.711238 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:49:55.559711 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:55.559663 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:49:55.639754 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:55.639690 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" event={"ID":"d2a6db99-8c3d-4882-bf3f-35a6269f9a71","Type":"ContainerStarted","Data":"654f420acd8338e9e037af97e05919de2b7eb3fea6818b48c9b1c960ca0373d8"} Nov 01 08:49:55.639938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:55.639780 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" event={"ID":"d2a6db99-8c3d-4882-bf3f-35a6269f9a71","Type":"ContainerStarted","Data":"0c4f9700e52a17e3c0898fe6d878d4e6147f89ebbb15c34c84e93c92084f5723"} Nov 01 08:49:55.836299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:55.836178 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podStartSLOduration=2.836163892 podStartE2EDuration="2.836163892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:49:55.806269504 +0000 UTC m=+1018.894606189" watchObservedRunningTime="2026-11-01 08:49:55.836163892 +0000 UTC m=+1018.924500560" Nov 01 08:49:56.271989 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:56.271947 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:49:56.273382 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:56.273343 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:49:56.273536 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:56.273408 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:49:57.272707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:57.272671 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:49:57.273165 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:57.272751 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:49:58.272677 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:58.272639 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:49:58.272896 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:58.272699 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:49:59.272257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:59.272216 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:49:59.272483 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:49:59.272287 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:50:00.272792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:00.272759 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:50:00.273295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:00.272817 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:50:01.271733 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:01.271694 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:50:01.271932 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:01.271788 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:50:02.272695 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:02.272656 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 08:50:02.273217 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:02.272759 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 08:50:04.158633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:04.158596 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:50:04.159163 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:04.159101 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:50:04.160048 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:04.160026 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 08:50:05.557638 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:05.557593 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:50:15.478136 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:50:15.478097 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:50:15.565859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:15.565802 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:50:16.619986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.619948 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp"] Nov 01 08:50:16.630157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.630129 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.679071 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.679013 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp"] Nov 01 08:50:16.722313 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.722281 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"distinct-w1-dfdd13-kserve-self-signed-certs\"" Nov 01 08:50:16.724174 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.724145 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.724328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.724197 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.724404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.724373 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkv5\" (UniqueName: \"kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.724454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.724424 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.724511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.724492 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.724615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.724602 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.826341 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.826293 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.827346 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.827318 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.827554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.827538 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.827746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.827709 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkv5\" (UniqueName: \"kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.827901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.827879 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.828068 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.828054 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.829029 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.829005 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.829392 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.829373 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.830347 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.830052 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.833585 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.833432 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.840884 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.840853 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.887554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.887468 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkv5\" (UniqueName: \"kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5\") pod \"distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:16.943348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:16.943305 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:17.392282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:17.392248 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp"] Nov 01 08:50:17.399958 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:50:17.399918 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ef41f20_0c0c_4283_812c_8141c9b3c733.slice/crio-2079d8b8f16b82b6d642f21f488165a8145e5439fac0e91c90e8bf54cb646c0d WatchSource:0}: Error finding container 2079d8b8f16b82b6d642f21f488165a8145e5439fac0e91c90e8bf54cb646c0d: Status 404 returned error can't find the container with id 2079d8b8f16b82b6d642f21f488165a8145e5439fac0e91c90e8bf54cb646c0d Nov 01 08:50:17.771985 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:17.771882 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" event={"ID":"0ef41f20-0c0c-4283-812c-8141c9b3c733","Type":"ContainerStarted","Data":"d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9"} Nov 01 08:50:17.771985 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:17.771928 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" event={"ID":"0ef41f20-0c0c-4283-812c-8141c9b3c733","Type":"ContainerStarted","Data":"2079d8b8f16b82b6d642f21f488165a8145e5439fac0e91c90e8bf54cb646c0d"} Nov 01 08:50:17.883384 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:17.883323 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podStartSLOduration=1.883305499 podStartE2EDuration="1.883305499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:17.881220748 +0000 UTC m=+1040.969557452" watchObservedRunningTime="2026-11-01 08:50:17.883305499 +0000 UTC m=+1040.971642171" Nov 01 08:50:17.943777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:17.943711 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:25.337229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.337196 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc"] Nov 01 08:50:25.338610 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.338574 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:50:25.373848 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.373816 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr"] Nov 01 08:50:25.377938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.377917 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.426779 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.426741 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-51b2b41abcdb\"" Nov 01 08:50:25.442001 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.441969 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr"] Nov 01 08:50:25.443250 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.443173 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.443460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.443337 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.443460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.443422 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5rj\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.445731 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.443521 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.544379 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.544317 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.544586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.544409 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5rj\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.544586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.544469 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.544586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.544508 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.545174 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.545156 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.545263 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.545243 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.547244 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.547196 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.603498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.603460 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5rj\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj\") pod \"payload-processing-external-model-b6fcd7955-xcvfr\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:25.692552 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:25.692507 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:26.069352 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.069268 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr"] Nov 01 08:50:26.077393 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:50:26.077360 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939769f2_5b71_460b_acd9_56389f649cb8.slice/crio-783fa01ac67e378c15141c714020f8d764c54052b1c92f345a2220d01d47d19a WatchSource:0}: Error finding container 783fa01ac67e378c15141c714020f8d764c54052b1c92f345a2220d01d47d19a: Status 404 returned error can't find the container with id 783fa01ac67e378c15141c714020f8d764c54052b1c92f345a2220d01d47d19a Nov 01 08:50:26.817375 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.817339 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:50:26.830129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.830093 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" event={"ID":"939769f2-5b71-460b-acd9-56389f649cb8","Type":"ContainerStarted","Data":"bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c"} Nov 01 08:50:26.830343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.830156 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" event={"ID":"939769f2-5b71-460b-acd9-56389f649cb8","Type":"ContainerStarted","Data":"783fa01ac67e378c15141c714020f8d764c54052b1c92f345a2220d01d47d19a"} Nov 01 08:50:26.830343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.830241 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:50:26.939550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.939481 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" podStartSLOduration=1.9394612260000001 podStartE2EDuration="1.939461226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:26.939124939 +0000 UTC m=+1050.027461611" watchObservedRunningTime="2026-11-01 08:50:26.939461226 +0000 UTC m=+1050.027797899" Nov 01 08:50:26.946782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:26.945940 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:27.005742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:27.004900 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:27.856049 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:27.856013 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:50:30.339882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:30.339801 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" containerID="cri-o://6d37421240030f24acc4ddf83eba56e32785e67aab0c05f1723ffbd4a29b10a7" gracePeriod=25 Nov 01 08:50:30.964025 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:30.963930 5710 generic.go:356] "Generic (PLEG): container finished" podID="e3db70da-51ec-4e7c-b12b-bff305405337" containerID="6d37421240030f24acc4ddf83eba56e32785e67aab0c05f1723ffbd4a29b10a7" exitCode=0 Nov 01 08:50:30.964450 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:30.964425 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" event={"ID":"e3db70da-51ec-4e7c-b12b-bff305405337","Type":"ContainerDied","Data":"6d37421240030f24acc4ddf83eba56e32785e67aab0c05f1723ffbd4a29b10a7"} Nov 01 08:50:30.964617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:30.964600 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" event={"ID":"e3db70da-51ec-4e7c-b12b-bff305405337","Type":"ContainerDied","Data":"ed52d244407f77115c56f0d8e4b1c18bb18eed7c9ea7d44c82e48133d8d84d63"} Nov 01 08:50:30.964756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:30.964739 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed52d244407f77115c56f0d8e4b1c18bb18eed7c9ea7d44c82e48133d8d84d63" Nov 01 08:50:30.976652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:30.976629 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:50:31.164757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.163839 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume\") pod \"e3db70da-51ec-4e7c-b12b-bff305405337\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " Nov 01 08:50:31.164757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.163968 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9\") pod \"e3db70da-51ec-4e7c-b12b-bff305405337\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " Nov 01 08:50:31.164757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.164128 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume\") pod \"e3db70da-51ec-4e7c-b12b-bff305405337\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " Nov 01 08:50:31.164757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.164206 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials\") pod \"e3db70da-51ec-4e7c-b12b-bff305405337\" (UID: \"e3db70da-51ec-4e7c-b12b-bff305405337\") " Nov 01 08:50:31.164757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.164360 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume" pod "e3db70da-51ec-4e7c-b12b-bff305405337" (UID: "e3db70da-51ec-4e7c-b12b-bff305405337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:50:31.165187 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.164803 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:50:31.165990 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.165958 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume" pod "e3db70da-51ec-4e7c-b12b-bff305405337" (UID: "e3db70da-51ec-4e7c-b12b-bff305405337"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:50:31.169661 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.169611 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials" pod "e3db70da-51ec-4e7c-b12b-bff305405337" (UID: "e3db70da-51ec-4e7c-b12b-bff305405337"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:50:31.170557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.170523 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9" pod "e3db70da-51ec-4e7c-b12b-bff305405337" (UID: "e3db70da-51ec-4e7c-b12b-bff305405337"). InnerVolumeSpecName "kube-api-access-hmvz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:50:31.268806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.265759 5710 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3db70da-51ec-4e7c-b12b-bff305405337-routing-overlay-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:50:31.268806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.265802 5710 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-provider-credentials\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:50:31.268806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.265818 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmvz9\" (UniqueName: \"kubernetes.io/projected/e3db70da-51ec-4e7c-b12b-bff305405337-kube-api-access-hmvz9\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:50:31.971062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:31.971030 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc" Nov 01 08:50:32.128644 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:32.128042 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc"] Nov 01 08:50:32.182866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:32.182529 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-74bd68c8cc-8xwxc"] Nov 01 08:50:33.058802 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:33.058764 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" path="/var/lib/kubelet/pods/e3db70da-51ec-4e7c-b12b-bff305405337/volumes" Nov 01 08:50:37.840750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:37.840696 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:50:45.477625 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:50:45.477582 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:50:47.783276 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.783218 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl"] Nov 01 08:50:47.783851 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.783833 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" Nov 01 08:50:47.783907 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.783853 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" Nov 01 08:50:47.783976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.783966 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3db70da-51ec-4e7c-b12b-bff305405337" containerName="payload-processing" Nov 01 08:50:47.787332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.787311 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.838963 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.838926 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:50:47.870988 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.870945 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.871328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.871305 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.871572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.871543 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfq6\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.871844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.871796 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.872033 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.872007 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.872158 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.872057 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.873138 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.873100 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.873267 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.873208 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.873370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.873281 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.875352 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.875325 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl"] Nov 01 08:50:47.974239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974198 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfq6\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974241 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974368 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974423 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974465 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974665 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974553 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974665 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974578 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974665 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974603 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974702 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974773 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974807 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.974863 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974856 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.975018 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.974982 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.975472 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.975410 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.977166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.977143 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:47.977345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:47.977328 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:48.033936 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.033848 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfq6\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:48.034276 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.034253 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:48.102734 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.102681 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:48.574603 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.574567 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl"] Nov 01 08:50:48.581921 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:50:48.581889 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a81c8f_9198_4d4b_b564_0c288147c8bb.slice/crio-cf4cd089663c19dffa64fd58f849baf3e3d8166f437b4b2b09889aaf267f4af9 WatchSource:0}: Error finding container cf4cd089663c19dffa64fd58f849baf3e3d8166f437b4b2b09889aaf267f4af9: Status 404 returned error can't find the container with id cf4cd089663c19dffa64fd58f849baf3e3d8166f437b4b2b09889aaf267f4af9 Nov 01 08:50:48.584428 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.584389 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:50:48.584553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.584487 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:50:48.584553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:48.584516 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:50:49.098312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:49.098259 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" event={"ID":"c2a81c8f-9198-4d4b-b564-0c288147c8bb","Type":"ContainerStarted","Data":"79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203"} Nov 01 08:50:49.099002 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:49.098975 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" event={"ID":"c2a81c8f-9198-4d4b-b564-0c288147c8bb","Type":"ContainerStarted","Data":"cf4cd089663c19dffa64fd58f849baf3e3d8166f437b4b2b09889aaf267f4af9"} Nov 01 08:50:49.107260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:49.107210 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:49.107758 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:49.107700 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:49.107977 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:49.107940 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:49.217510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:49.217457 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podStartSLOduration=2.217441394 podStartE2EDuration="2.217441394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:50:49.216584836 +0000 UTC m=+1072.304921508" watchObservedRunningTime="2026-11-01 08:50:49.217441394 +0000 UTC m=+1072.305778065" Nov 01 08:50:50.105783 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:50.104845 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:50.105783 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:50.104929 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:51.106160 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:51.106122 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:51.106572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:51.106202 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:52.105745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:52.104133 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:52.105745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:52.104211 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:53.106201 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:53.106157 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:53.106623 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:53.106221 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:54.105749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:54.104153 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:54.105749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:54.104222 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:55.103973 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:55.103936 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 08:50:55.104386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:55.103990 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 08:50:56.109061 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:56.109031 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:56.109488 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:56.109419 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:56.110162 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:56.110145 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 08:50:56.113994 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:56.113972 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:50:57.839055 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:57.839013 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:50:59.013974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.013940 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd"] Nov 01 08:50:59.019754 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.019733 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.150098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.149513 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd"] Nov 01 08:50:59.249183 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249135 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249183 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249181 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249416 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249397 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249514 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249500 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkkp\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249582 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249702 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249682 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249829 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249860 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.249885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.249879 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.350916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.350882 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.350973 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkkp\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351076 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351107 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351186 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351161 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351207 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351249 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351294 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351317 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351389 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351459 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351582 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351801 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351768 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.351990 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.351972 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.357556 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.357492 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.360356 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.360309 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.408519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.408478 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.408708 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.408595 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkkp\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:50:59.637857 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:50:59.637776 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:51:00.072747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.072657 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd"] Nov 01 08:51:00.079511 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:51:00.079473 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612f4800_78b3_49ff_9b00_41415daa0fea.slice/crio-ce794b9cadde389379eb2e5caade500cfacf0cd20c30d179f0169b3af8dfd988 WatchSource:0}: Error finding container ce794b9cadde389379eb2e5caade500cfacf0cd20c30d179f0169b3af8dfd988: Status 404 returned error can't find the container with id ce794b9cadde389379eb2e5caade500cfacf0cd20c30d179f0169b3af8dfd988 Nov 01 08:51:00.082169 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.082135 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:00.082268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.082212 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:00.082268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.082243 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:00.183489 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.183454 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" event={"ID":"612f4800-78b3-49ff-9b00-41415daa0fea","Type":"ContainerStarted","Data":"31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2"} Nov 01 08:51:00.183709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.183509 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" event={"ID":"612f4800-78b3-49ff-9b00-41415daa0fea","Type":"ContainerStarted","Data":"ce794b9cadde389379eb2e5caade500cfacf0cd20c30d179f0169b3af8dfd988"} Nov 01 08:51:00.298248 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.298190 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podStartSLOduration=2.298172814 podStartE2EDuration="2.298172814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:00.296618552 +0000 UTC m=+1083.384955226" watchObservedRunningTime="2026-11-01 08:51:00.298172814 +0000 UTC m=+1083.386509485" Nov 01 08:51:00.638582 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:00.638546 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:51:01.640159 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:01.640117 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 08:51:01.640595 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:01.640172 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 08:51:02.162778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.162736 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr"] Nov 01 08:51:02.164468 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.164436 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:51:02.206945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.206900 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn"] Nov 01 08:51:02.212279 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.212249 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.217670 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.217633 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.217903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.217821 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.217978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.217930 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.217978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.217970 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9z4\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.261212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.261168 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-078fe17fccfe\"" Nov 01 08:51:02.261212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.261183 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-951c92a637ed\"" Nov 01 08:51:02.268360 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.268324 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn"] Nov 01 08:51:02.319566 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.319519 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.319843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.319654 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.319944 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.319893 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.320009 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.319959 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9z4\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.320499 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.320477 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.320651 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.320626 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.323222 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.323196 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.377901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.377863 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9z4\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4\") pod \"payload-processing-external-model-c69ddc68f-xd8xn\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.531644 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.531544 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:02.572983 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.572948 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj"] Nov 01 08:51:02.612746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.612612 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.624513 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.624552 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.624589 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.624650 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.624818 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jh8\" (UniqueName: \"kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.624849 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.631935 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.631541 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj"] Nov 01 08:51:02.638562 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.638404 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 08:51:02.638562 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.638460 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 08:51:02.661808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.661764 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"openshift-service-ca.crt\"" Nov 01 08:51:02.661808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.661771 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"kube-root-ca.crt\"" Nov 01 08:51:02.664388 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.664360 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"default-dockercfg-fp8xg\"" Nov 01 08:51:02.664543 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.664429 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-54e01b\"/\"e2e-conflict-b-ef480be9-kserve-self-signed-certs\"" Nov 01 08:51:02.732601 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.732544 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.733447 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.733301 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.733941 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.733416 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.734157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.733915 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.734319 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.734299 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.734853 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.734741 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.735462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.734996 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.735793 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.735772 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jh8\" (UniqueName: \"kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.835836 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.743849 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.835836 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.832555 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jh8\" (UniqueName: \"kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.836672 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.836602 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.837600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.837572 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs\") pod \"e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.931386 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.930984 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:02.972502 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:02.956156 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn"] Nov 01 08:51:03.236874 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.236833 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" event={"ID":"0725d742-71e3-46ba-93b7-5f31c1671081","Type":"ContainerStarted","Data":"7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013"} Nov 01 08:51:03.237112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.236896 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" event={"ID":"0725d742-71e3-46ba-93b7-5f31c1671081","Type":"ContainerStarted","Data":"4a27f60e16701e315ae3558bda1b001a0c3d086d45ef66c82db559ead363a551"} Nov 01 08:51:03.237112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.236924 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:03.355701 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.355625 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" podStartSLOduration=1.355609656 podStartE2EDuration="1.355609656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:03.35394267 +0000 UTC m=+1086.442279341" watchObservedRunningTime="2026-11-01 08:51:03.355609656 +0000 UTC m=+1086.443946329" Nov 01 08:51:03.458107 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.457290 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj"] Nov 01 08:51:03.464744 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:51:03.464695 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf042b65_098b_45a7_a8f5_29704a28fb65.slice/crio-4386f360e8404312d03f364c8742498429bd11605e48748eef264ad05af51ad7 WatchSource:0}: Error finding container 4386f360e8404312d03f364c8742498429bd11605e48748eef264ad05af51ad7: Status 404 returned error can't find the container with id 4386f360e8404312d03f364c8742498429bd11605e48748eef264ad05af51ad7 Nov 01 08:51:03.640415 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.640377 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 08:51:03.640619 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:03.640444 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 08:51:04.245032 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:04.244948 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" event={"ID":"df042b65-098b-45a7-a8f5-29704a28fb65","Type":"ContainerStarted","Data":"fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34"} Nov 01 08:51:04.245032 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:04.244990 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" event={"ID":"df042b65-098b-45a7-a8f5-29704a28fb65","Type":"ContainerStarted","Data":"4386f360e8404312d03f364c8742498429bd11605e48748eef264ad05af51ad7"} Nov 01 08:51:04.359504 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:04.359438 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" podStartSLOduration=2.3594105020000002 podStartE2EDuration="2.359410502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:04.355873833 +0000 UTC m=+1087.444210517" watchObservedRunningTime="2026-11-01 08:51:04.359410502 +0000 UTC m=+1087.447747175" Nov 01 08:51:04.638145 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:04.638109 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 08:51:04.638354 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:04.638163 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 08:51:04.947809 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:04.944827 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:05.648121 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:05.648080 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 08:51:05.648672 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:05.648145 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 08:51:06.638551 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:06.638503 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 08:51:06.640411 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:06.638888 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 08:51:07.168753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.164871 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" containerID="cri-o://bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c" gracePeriod=25 Nov 01 08:51:07.514515 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.514477 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:51:07.624098 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.624066 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials\") pod \"939769f2-5b71-460b-acd9-56389f649cb8\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " Nov 01 08:51:07.624299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.624137 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj\") pod \"939769f2-5b71-460b-acd9-56389f649cb8\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " Nov 01 08:51:07.624299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.624243 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume\") pod \"939769f2-5b71-460b-acd9-56389f649cb8\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " Nov 01 08:51:07.624299 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.624263 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume\") pod \"939769f2-5b71-460b-acd9-56389f649cb8\" (UID: \"939769f2-5b71-460b-acd9-56389f649cb8\") " Nov 01 08:51:07.625002 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.624975 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume" pod "939769f2-5b71-460b-acd9-56389f649cb8" (UID: "939769f2-5b71-460b-acd9-56389f649cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:51:07.625834 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.625598 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume" pod "939769f2-5b71-460b-acd9-56389f649cb8" (UID: "939769f2-5b71-460b-acd9-56389f649cb8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:51:07.639820 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.639776 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials" pod "939769f2-5b71-460b-acd9-56389f649cb8" (UID: "939769f2-5b71-460b-acd9-56389f649cb8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:51:07.640955 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.640917 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj" pod "939769f2-5b71-460b-acd9-56389f649cb8" (UID: "939769f2-5b71-460b-acd9-56389f649cb8"). InnerVolumeSpecName "kube-api-access-9k5rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:51:07.726553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.726486 5710 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-routing-overlay-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:51:07.726553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.726523 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939769f2-5b71-460b-acd9-56389f649cb8-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:51:07.726553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.726539 5710 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-provider-credentials\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:51:07.726929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:07.726567 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k5rj\" (UniqueName: \"kubernetes.io/projected/939769f2-5b71-460b-acd9-56389f649cb8-kube-api-access-9k5rj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:51:08.036081 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.036045 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:51:08.037760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.037732 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:51:08.038900 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.038876 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 08:51:08.292430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.292308 5710 generic.go:356] "Generic (PLEG): container finished" podID="939769f2-5b71-460b-acd9-56389f649cb8" containerID="bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c" exitCode=0 Nov 01 08:51:08.292430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.292372 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" Nov 01 08:51:08.292430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.292388 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" event={"ID":"939769f2-5b71-460b-acd9-56389f649cb8","Type":"ContainerDied","Data":"bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c"} Nov 01 08:51:08.293063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.292462 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr" event={"ID":"939769f2-5b71-460b-acd9-56389f649cb8","Type":"ContainerDied","Data":"783fa01ac67e378c15141c714020f8d764c54052b1c92f345a2220d01d47d19a"} Nov 01 08:51:08.293063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.292493 5710 scope.go:122] "RemoveContainer" containerID="bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c" Nov 01 08:51:08.306297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.306274 5710 scope.go:122] "RemoveContainer" containerID="bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c" Nov 01 08:51:08.306684 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:51:08.306661 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c\": container with ID starting with bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c not found: ID does not exist" containerID="bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c" Nov 01 08:51:08.306783 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.306695 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c"} err="failed to get container status \"bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c\": rpc error: code = NotFound desc = could not find container \"bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c\": container with ID starting with bbb4ded5dfb99da0ad92a5d649ce07d21ca59a1bb9ed0bdcea68681b3d14897c not found: ID does not exist" Nov 01 08:51:08.447495 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.447455 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr"] Nov 01 08:51:08.451688 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:08.451655 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-xcvfr"] Nov 01 08:51:09.058022 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:09.057987 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939769f2-5b71-460b-acd9-56389f649cb8" path="/var/lib/kubelet/pods/939769f2-5b71-460b-acd9-56389f649cb8/volumes" Nov 01 08:51:12.932048 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:12.932002 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:12.944482 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:12.944454 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:13.333301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:13.333216 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:51:14.250738 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:14.249932 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:51:15.490299 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:51:15.490220 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:51:17.255746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:17.252844 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:51:17.255746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:17.252934 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:51:25.335410 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:25.335372 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:51:41.306788 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.306757 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc"] Nov 01 08:51:41.309191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.307341 5710 cpu_manager.go:414] "containerMap: removing container" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" Nov 01 08:51:41.309191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.307360 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" Nov 01 08:51:41.309191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.307504 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="939769f2-5b71-460b-acd9-56389f649cb8" containerName="payload-processing" Nov 01 08:51:41.310804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.310789 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.408500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.408464 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfdd13\"/\"distinct-2-w1-dfdd13-kserve-self-signed-certs\"" Nov 01 08:51:41.410966 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.410932 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc"] Nov 01 08:51:41.488752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.488197 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.488752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.488625 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.493748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.489545 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.493748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.489827 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7266l\" (UniqueName: \"kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.493748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.489861 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.493748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.489938 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.590781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.590712 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.590962 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.590942 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7266l\" (UniqueName: \"kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.590980 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.591048 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591127 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.591110 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.591155 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591265 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.591159 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.591522 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.591554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.591543 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.593572 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.593543 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.593811 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.593792 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.648728 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.648677 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7266l\" (UniqueName: \"kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l\") pod \"distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.794576 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.794541 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd"] Nov 01 08:51:41.799534 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.799510 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.827261 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.827205 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc"] Nov 01 08:51:41.831638 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.831610 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.859980 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.859894 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd"] Nov 01 08:51:41.898210 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898177 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.898210 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898214 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898282 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.898460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898315 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898334 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898357 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.898460 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898372 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898497 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9v6\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898633 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.898801 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898672 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.898801 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898761 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898821 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.898903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898875 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.898903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898893 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.899035 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.898915 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6m6x\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.899092 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.899049 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:41.899153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.899109 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.899234 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.899160 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:41.923756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.923695 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:41.945065 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:41.945033 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc"] Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.004710 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.004808 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.004855 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.004903 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.004974 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.004999 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.005027 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6m6x\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.005546 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.005060 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.006119 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.005913 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.006252 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006224 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.006597 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006574 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.006683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006667 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.006777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006760 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.006845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006791 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.006845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006821 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.006950 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006863 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.006950 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006891 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.006950 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006928 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.007104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006952 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.007104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.006983 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9v6\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.015397 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.015360 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.017485 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.017449 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.019013 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.018981 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.019154 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.019017 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.020348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.019318 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.020348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.019579 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.020348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.019834 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.020348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.019943 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.020348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.019978 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.025034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.021883 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.025034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.024561 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.025034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.024734 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.077900 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.077857 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.077900 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.077892 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.078166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.077979 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9v6\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.078270 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.078249 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6m6x\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.113610 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.113509 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:42.146323 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.146282 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:42.314880 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.314843 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc"] Nov 01 08:51:42.321601 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:51:42.321564 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ff0e23_0b62_4c33_97f7_14357d8675cf.slice/crio-d60ca322f5c4363ae946d21469ba2403434e5490eca2e9e4ad3bf336897be145 WatchSource:0}: Error finding container d60ca322f5c4363ae946d21469ba2403434e5490eca2e9e4ad3bf336897be145: Status 404 returned error can't find the container with id d60ca322f5c4363ae946d21469ba2403434e5490eca2e9e4ad3bf336897be145 Nov 01 08:51:42.496124 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.496087 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd"] Nov 01 08:51:42.501305 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:51:42.501271 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bacc7b_11be_442c_b337_815ac913ffe3.slice/crio-21cc09ea209a9b1df962e79a01e46b2ed8d5f18fe78b44b621b05ba9e370dd1a WatchSource:0}: Error finding container 21cc09ea209a9b1df962e79a01e46b2ed8d5f18fe78b44b621b05ba9e370dd1a: Status 404 returned error can't find the container with id 21cc09ea209a9b1df962e79a01e46b2ed8d5f18fe78b44b621b05ba9e370dd1a Nov 01 08:51:42.503965 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.503927 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:42.504077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.504017 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:42.504077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.504063 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:42.545651 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.545624 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc"] Nov 01 08:51:42.546521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.546487 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" event={"ID":"b3ff0e23-0b62-4c33-97f7-14357d8675cf","Type":"ContainerStarted","Data":"8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585"} Nov 01 08:51:42.546676 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.546536 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" event={"ID":"b3ff0e23-0b62-4c33-97f7-14357d8675cf","Type":"ContainerStarted","Data":"d60ca322f5c4363ae946d21469ba2403434e5490eca2e9e4ad3bf336897be145"} Nov 01 08:51:42.548294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.548267 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" event={"ID":"a2bacc7b-11be-442c-b337-815ac913ffe3","Type":"ContainerStarted","Data":"21cc09ea209a9b1df962e79a01e46b2ed8d5f18fe78b44b621b05ba9e370dd1a"} Nov 01 08:51:42.558875 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:51:42.558847 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d595622_6926_49df_b3f9_682fa21c727d.slice/crio-04a1828429c1709b76da2443331ed1f0b92bc480f2526edf326a8df791b5e659 WatchSource:0}: Error finding container 04a1828429c1709b76da2443331ed1f0b92bc480f2526edf326a8df791b5e659: Status 404 returned error can't find the container with id 04a1828429c1709b76da2443331ed1f0b92bc480f2526edf326a8df791b5e659 Nov 01 08:51:42.561275 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.561245 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:42.561385 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.561330 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:42.561441 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.561398 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:51:42.658878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.658816 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podStartSLOduration=1.658796539 podStartE2EDuration="1.658796539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:42.657051538 +0000 UTC m=+1125.745388210" watchObservedRunningTime="2026-11-01 08:51:42.658796539 +0000 UTC m=+1125.747133216" Nov 01 08:51:42.924115 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:42.923984 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:43.556963 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:43.556921 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" event={"ID":"a2bacc7b-11be-442c-b337-815ac913ffe3","Type":"ContainerStarted","Data":"4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81"} Nov 01 08:51:43.558730 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:43.558688 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" event={"ID":"4d595622-6926-49df-b3f9-682fa21c727d","Type":"ContainerStarted","Data":"332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046"} Nov 01 08:51:43.558876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:43.558747 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" event={"ID":"4d595622-6926-49df-b3f9-682fa21c727d","Type":"ContainerStarted","Data":"04a1828429c1709b76da2443331ed1f0b92bc480f2526edf326a8df791b5e659"} Nov 01 08:51:43.672844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:43.672786 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podStartSLOduration=2.672760542 podStartE2EDuration="2.672760542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:43.669479408 +0000 UTC m=+1126.757816080" watchObservedRunningTime="2026-11-01 08:51:43.672760542 +0000 UTC m=+1126.761097211" Nov 01 08:51:43.811805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:43.811087 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podStartSLOduration=2.811062862 podStartE2EDuration="2.811062862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:51:43.803059489 +0000 UTC m=+1126.891396161" watchObservedRunningTime="2026-11-01 08:51:43.811062862 +0000 UTC m=+1126.899399534" Nov 01 08:51:44.117879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:44.117778 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:44.118563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:44.118190 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 08:51:44.118563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:44.118248 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 08:51:44.146553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:44.146517 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:44.147821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:44.147798 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 08:51:44.147937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:44.147850 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 08:51:45.116743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:45.116180 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 08:51:45.116743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:45.116244 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 08:51:45.147628 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:45.147594 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 08:51:45.147938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:45.147909 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 08:51:45.478255 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:51:45.478168 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:51:46.115560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:46.115087 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 08:51:46.115560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:46.115147 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 08:51:46.147400 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:46.147362 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 08:51:46.147859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:46.147452 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 08:51:47.123744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:47.122878 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 08:51:47.123744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:47.122951 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 08:51:47.147327 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:47.147286 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 08:51:47.147526 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:47.147346 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 08:51:48.114742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:48.114233 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 08:51:48.114742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:48.114300 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 08:51:48.147292 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:48.147253 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 08:51:48.147469 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:48.147321 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 08:51:49.114269 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:49.114225 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 08:51:49.114704 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:49.114287 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 08:51:49.146881 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:49.146836 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 08:51:49.147058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:49.146908 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 08:51:50.068469 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:50.068421 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:51:50.893741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:50.892921 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:50.917741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:50.894452 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:50.917741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:50.896915 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 08:51:51.120081 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:51.120048 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:51.121052 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:51.121026 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:51.135962 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:51.135936 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 08:51:51.926619 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:51.926578 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:51.946455 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:51.946412 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:52.741258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:52.741228 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:51:57.294507 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:51:57.294473 5710 scope.go:122] "RemoveContainer" containerID="896b5ad5ef1242a1b098845f1ab67c048f70c78a1eb4e5f6f8f970929a87155e" Nov 01 08:52:15.478061 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:52:15.478021 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:52:18.497111 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.497073 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8"] Nov 01 08:52:18.501561 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.501532 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543342 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543300 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543342 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543345 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543583 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543483 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwbg\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543583 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543558 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543647 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543602 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543681 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543656 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543681 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543674 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543793 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.543868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.543855 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.620295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.620257 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8"] Nov 01 08:52:18.644581 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644546 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.644859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644645 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.644859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644671 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.644859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644775 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwbg\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.644859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644829 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.645104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644871 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.645104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644910 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.645104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644933 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.645104 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.644985 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.645709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.645670 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.645709 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.645691 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.646107 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.645813 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.646107 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.645975 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.646107 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.646040 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.647768 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.647743 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.649466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.649444 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.702130 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.702096 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.702564 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.702541 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwbg\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:18.816272 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:18.816166 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:19.255390 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:19.255348 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8"] Nov 01 08:52:19.262026 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:52:19.261995 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7259d009_0b44_423b_b235_2ded03c6b9e0.slice/crio-debc48829a1320207374743ccc25f72c8fa04fd75f84b7d130a99e97ed2e3c66 WatchSource:0}: Error finding container debc48829a1320207374743ccc25f72c8fa04fd75f84b7d130a99e97ed2e3c66: Status 404 returned error can't find the container with id debc48829a1320207374743ccc25f72c8fa04fd75f84b7d130a99e97ed2e3c66 Nov 01 08:52:19.264751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:19.264687 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:19.264846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:19.264783 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:19.264846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:19.264816 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:19.918545 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:19.918507 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" event={"ID":"7259d009-0b44-423b-b235-2ded03c6b9e0","Type":"ContainerStarted","Data":"38f999c9c319caaaadc01db4b2469dbfd77e6297ac3143054a6fc7e41ddf4633"} Nov 01 08:52:19.919332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:19.919303 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" event={"ID":"7259d009-0b44-423b-b235-2ded03c6b9e0","Type":"ContainerStarted","Data":"debc48829a1320207374743ccc25f72c8fa04fd75f84b7d130a99e97ed2e3c66"} Nov 01 08:52:20.079991 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:20.079871 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podStartSLOduration=2.079853821 podStartE2EDuration="2.079853821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:20.078943251 +0000 UTC m=+1163.167279935" watchObservedRunningTime="2026-11-01 08:52:20.079853821 +0000 UTC m=+1163.168190574" Nov 01 08:52:20.816881 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:20.816835 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:20.818306 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:20.818281 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 08:52:20.818389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:20.818367 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 08:52:21.817096 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:21.817058 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 08:52:21.817544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:21.817145 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 08:52:22.817518 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:22.817474 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 08:52:22.817986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:22.817542 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 08:52:23.817022 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:23.816979 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 08:52:23.817259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:23.817060 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 08:52:24.816664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:24.816623 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 08:52:24.817123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:24.816679 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 08:52:25.050132 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:25.050096 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj"] Nov 01 08:52:25.817084 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:25.817039 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 08:52:25.817498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:25.817124 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 08:52:27.324340 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:27.324297 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:27.325359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:27.325334 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:27.329349 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:27.326981 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 08:52:29.261119 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.261075 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2"] Nov 01 08:52:29.265815 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.265788 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.378937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.378872 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2"] Nov 01 08:52:29.438021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.437979 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438111 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438292 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438266 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438364 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438345 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbm86\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438461 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438441 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438527 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438509 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438576 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438636 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438622 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.438685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.438654 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540432 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbm86\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540490 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540785 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540665 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540785 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540759 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540898 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540794 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540924 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.540953 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.540935 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541062 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541044 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541142 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541125 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541142 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541133 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541291 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541275 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541436 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541415 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541492 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.541569 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.541553 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.543490 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.543459 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.543668 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.543644 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.597998 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.597963 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.598163 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.597999 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbm86\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:29.881099 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:29.881062 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:30.313877 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:30.313786 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2"] Nov 01 08:52:30.358885 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:52:30.358845 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b6f90f_bf37_48be_aeba_1e6ef73b373b.slice/crio-b24db8bccc276bd87d3a2b5759c7416d045fbc4257c69597d51f7ca37619d09e WatchSource:0}: Error finding container b24db8bccc276bd87d3a2b5759c7416d045fbc4257c69597d51f7ca37619d09e: Status 404 returned error can't find the container with id b24db8bccc276bd87d3a2b5759c7416d045fbc4257c69597d51f7ca37619d09e Nov 01 08:52:30.361426 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:30.361394 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:30.361509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:30.361467 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:30.361509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:30.361500 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:31.012846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.012810 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" event={"ID":"82b6f90f-bf37-48be-aeba-1e6ef73b373b","Type":"ContainerStarted","Data":"08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158"} Nov 01 08:52:31.013072 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.012864 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" event={"ID":"82b6f90f-bf37-48be-aeba-1e6ef73b373b","Type":"ContainerStarted","Data":"b24db8bccc276bd87d3a2b5759c7416d045fbc4257c69597d51f7ca37619d09e"} Nov 01 08:52:31.206755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.206112 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podStartSLOduration=2.206091141 podStartE2EDuration="2.206091141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:31.171422322 +0000 UTC m=+1174.259758997" watchObservedRunningTime="2026-11-01 08:52:31.206091141 +0000 UTC m=+1174.294427815" Nov 01 08:52:31.431071 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.431030 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:52:31.899750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.898839 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:31.905845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.901996 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 08:52:31.905845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:31.902081 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 08:52:32.887219 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:32.887166 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 08:52:32.887906 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:32.887867 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 08:52:33.887748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:33.881585 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 08:52:33.887748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:33.881676 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 08:52:34.887976 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:34.887936 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 08:52:34.888600 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:34.888564 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 08:52:35.902743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:35.901122 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 08:52:35.902743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:35.901185 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 08:52:36.881937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:36.881895 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 08:52:36.882149 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:36.881966 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 08:52:38.371248 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:38.371205 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:38.372216 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:38.372194 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:38.373180 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:38.373155 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 08:52:40.071990 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.071938 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" podUID="df042b65-098b-45a7-a8f5-29704a28fb65" containerName="main" containerID="cri-o://fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34" gracePeriod=45 Nov 01 08:52:40.368067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.368040 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:52:40.419648 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419618 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir\") pod \"df042b65-098b-45a7-a8f5-29704a28fb65\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " Nov 01 08:52:40.419924 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419760 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm\") pod \"df042b65-098b-45a7-a8f5-29704a28fb65\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " Nov 01 08:52:40.419924 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419819 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs\" (UniqueName: \"kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs\") pod \"df042b65-098b-45a7-a8f5-29704a28fb65\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " Nov 01 08:52:40.419924 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419881 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8\" (UniqueName: \"kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8\") pod \"df042b65-098b-45a7-a8f5-29704a28fb65\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " Nov 01 08:52:40.420136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419931 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache\") pod \"df042b65-098b-45a7-a8f5-29704a28fb65\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " Nov 01 08:52:40.420136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419976 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home\") pod \"df042b65-098b-45a7-a8f5-29704a28fb65\" (UID: \"df042b65-098b-45a7-a8f5-29704a28fb65\") " Nov 01 08:52:40.420136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.419973 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir" pod "df042b65-098b-45a7-a8f5-29704a28fb65" (UID: "df042b65-098b-45a7-a8f5-29704a28fb65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:52:40.420297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.420230 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home" pod "df042b65-098b-45a7-a8f5-29704a28fb65" (UID: "df042b65-098b-45a7-a8f5-29704a28fb65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:52:40.420297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.420245 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache" pod "df042b65-098b-45a7-a8f5-29704a28fb65" (UID: "df042b65-098b-45a7-a8f5-29704a28fb65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:52:40.420778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.420756 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:52:40.420866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.420783 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:52:40.420866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.420799 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:52:40.422344 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.422319 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm" pod "df042b65-098b-45a7-a8f5-29704a28fb65" (UID: "df042b65-098b-45a7-a8f5-29704a28fb65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:52:40.422344 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.422334 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs" pod "df042b65-098b-45a7-a8f5-29704a28fb65" (UID: "df042b65-098b-45a7-a8f5-29704a28fb65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:52:40.422490 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.422344 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8" pod "df042b65-098b-45a7-a8f5-29704a28fb65" (UID: "df042b65-098b-45a7-a8f5-29704a28fb65"). InnerVolumeSpecName "kube-api-access-m8jh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:52:40.522370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.522328 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df042b65-098b-45a7-a8f5-29704a28fb65-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:52:40.522370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.522368 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df042b65-098b-45a7-a8f5-29704a28fb65-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:52:40.522596 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:40.522386 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jh8\" (UniqueName: \"kubernetes.io/projected/df042b65-098b-45a7-a8f5-29704a28fb65-kube-api-access-m8jh8\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:52:41.094956 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.094916 5710 generic.go:356] "Generic (PLEG): container finished" podID="df042b65-098b-45a7-a8f5-29704a28fb65" containerID="fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34" exitCode=0 Nov 01 08:52:41.095417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.094971 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" event={"ID":"df042b65-098b-45a7-a8f5-29704a28fb65","Type":"ContainerDied","Data":"fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34"} Nov 01 08:52:41.095417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.094980 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" Nov 01 08:52:41.095417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.095017 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" event={"ID":"df042b65-098b-45a7-a8f5-29704a28fb65","Type":"ContainerDied","Data":"4386f360e8404312d03f364c8742498429bd11605e48748eef264ad05af51ad7"} Nov 01 08:52:41.095417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.095038 5710 scope.go:122] "RemoveContainer" containerID="fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34" Nov 01 08:52:41.105477 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.105451 5710 scope.go:122] "RemoveContainer" containerID="fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34" Nov 01 08:52:41.105840 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:52:41.105810 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34\": container with ID starting with fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34 not found: ID does not exist" containerID="fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34" Nov 01 08:52:41.105923 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.105855 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34"} err="failed to get container status \"fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34\": rpc error: code = NotFound desc = could not find container \"fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34\": container with ID starting with fd0e24b5a64307e17a52f40a3ab0425033e9a3737551b1273b40dc5f3a279f34 not found: ID does not exist" Nov 01 08:52:41.248621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.248584 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj"] Nov 01 08:52:41.294986 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.294948 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj"] Nov 01 08:52:41.323583 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:41.323482 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-54e01b/e2e-conflict-b-ef480be9-kserve-7fc44d5db8-5hcmj" podUID="df042b65-098b-45a7-a8f5-29704a28fb65" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Nov 01 08:52:43.059003 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:43.058973 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df042b65-098b-45a7-a8f5-29704a28fb65" path="/var/lib/kubelet/pods/df042b65-098b-45a7-a8f5-29704a28fb65/volumes" Nov 01 08:52:45.478643 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:52:45.478601 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:52:53.056745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.038610 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p"] Nov 01 08:52:53.056745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.040149 5710 cpu_manager.go:414] "containerMap: removing container" podUID="df042b65-098b-45a7-a8f5-29704a28fb65" containerName="main" Nov 01 08:52:53.056745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.040259 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df042b65-098b-45a7-a8f5-29704a28fb65" containerName="main" Nov 01 08:52:53.056745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.051013 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="df042b65-098b-45a7-a8f5-29704a28fb65" containerName="main" Nov 01 08:52:53.096879 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.086322 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.126633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.126597 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p"] Nov 01 08:52:53.166972 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.166828 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.167417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.167377 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.167755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.167709 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.169437 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.169404 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.169771 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.169749 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.170055 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.170013 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.170314 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.170295 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fdc\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.170544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.170524 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.170828 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.170806 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274007 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274081 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274120 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274163 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274243 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fdc\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274331 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274417 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274507 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.274557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.274535 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.275281 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.275252 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.275603 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.275582 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.276157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.276128 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.276555 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.276531 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.276815 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.276790 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.279040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.279012 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.280043 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.279990 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.336878 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.336755 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fdc\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.337303 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.337266 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:53.439368 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:53.439018 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:54.665759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:54.665694 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:52:54.856757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:54.845812 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:54.856757 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:52:54.846260 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c022b90_56fc_4aeb_9a49_27ae002603f6.slice/crio-127906acccc491778e03c213ac542bda1facf711ab2278b604e751d1a560161f WatchSource:0}: Error finding container 127906acccc491778e03c213ac542bda1facf711ab2278b604e751d1a560161f: Status 404 returned error can't find the container with id 127906acccc491778e03c213ac542bda1facf711ab2278b604e751d1a560161f Nov 01 08:52:54.876777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:54.871221 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:54.876777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:54.871339 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:54.876777 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:54.871383 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:52:54.934868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:54.926963 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p"] Nov 01 08:52:55.284946 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:52:55.284889 5710 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c022b90_56fc_4aeb_9a49_27ae002603f6.slice/crio-7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304.scope\": RecentStats: unable to find data in memory cache]" Nov 01 08:52:55.673934 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:55.673871 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:55.708942 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:55.708883 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:52:55.795821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:55.795687 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" event={"ID":"3c022b90-56fc-4aeb-9a49-27ae002603f6","Type":"ContainerStarted","Data":"127906acccc491778e03c213ac542bda1facf711ab2278b604e751d1a560161f"} Nov 01 08:52:55.796222 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:55.796186 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:52:55.875566 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:55.875515 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:55.876261 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:55.876238 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:52:56.678775 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:56.676824 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:56.831493 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:56.831421 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:56.832105 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:56.832078 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:52:56.859994 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:56.859938 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" event={"ID":"3c022b90-56fc-4aeb-9a49-27ae002603f6","Type":"ContainerStarted","Data":"7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304"} Nov 01 08:52:56.877096 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:56.877054 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:57.136464 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.136428 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:52:57.143880 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.143813 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:57.475806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.464266 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:52:57.475806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.464327 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:52:57.596745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.592597 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podStartSLOduration=5.592576826 podStartE2EDuration="5.592576826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:52:57.386687777 +0000 UTC m=+1200.475024449" watchObservedRunningTime="2026-11-01 08:52:57.592576826 +0000 UTC m=+1200.680913499" Nov 01 08:52:57.699644 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.699597 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:57.852775 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.852470 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:57.918122 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.915778 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:57.945773 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:57.945499 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:58.306120 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:52:58.306085 5710 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.248s" Nov 01 08:52:58.306494 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.306477 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:52:58.417746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.415342 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:52:58.417746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.415418 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:58.443608 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.443355 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:52:58.443608 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.443419 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:52:58.723810 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.707403 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:52:58.875747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.853893 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:58.940756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:58.919169 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 08:52:59.446337 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:59.446293 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:52:59.446641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:59.446606 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:52:59.708825 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:59.707807 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:52:59.861912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:59.861861 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:52:59.960127 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:52:59.960020 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:00.298840 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.287047 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:00.578799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.556578 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:53:00.578799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.556643 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:53:00.642462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.640304 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:00.739589 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.739508 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:53:00.745198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.745167 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:53:00.754557 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:00.754523 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:53:01.440596 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:01.440549 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:53:01.441063 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:01.440614 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:53:02.441198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:02.441156 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:53:02.441827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:02.441792 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:53:03.440345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:03.440305 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:53:03.440642 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:03.440379 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:53:04.441004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:04.440968 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:53:04.441431 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:04.441037 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:53:05.440401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:05.440360 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 01 08:53:05.440616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:05.440422 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 01 08:53:07.085008 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:07.084960 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:53:07.086046 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:07.086013 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:53:07.088342 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:07.087849 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 08:53:08.140511 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.140466 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6"] Nov 01 08:53:08.146369 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.146338 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184234 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184202 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184272 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chswd\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184342 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184392 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184564 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184475 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184564 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184510 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184637 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184584 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184687 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184673 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.184753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.184741 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.262657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.262611 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6"] Nov 01 08:53:08.286434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286395 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chswd\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286434 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286440 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286466 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286503 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286522 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286566 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286703 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286691 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286973 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286790 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286973 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286845 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.286973 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.286941 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.287081 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.287056 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.287294 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.287272 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.287430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.287303 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.287616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.287532 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.289244 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.289220 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.289390 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.289369 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.350194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.350150 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.350360 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.350267 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chswd\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.462369 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.462273 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:08.895157 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.895112 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6"] Nov 01 08:53:08.902145 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:53:08.902112 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec208b0_7f97_4f99_a450_658f6de328bf.slice/crio-021e1cab253f2ebe0f9a1be3cf528677860f680914a25c89cba7b87b06731c0a WatchSource:0}: Error finding container 021e1cab253f2ebe0f9a1be3cf528677860f680914a25c89cba7b87b06731c0a: Status 404 returned error can't find the container with id 021e1cab253f2ebe0f9a1be3cf528677860f680914a25c89cba7b87b06731c0a Nov 01 08:53:08.904657 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.904626 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:08.904761 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.904704 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:08.904761 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:08.904754 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:09.662633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:09.662592 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" event={"ID":"1ec208b0-7f97-4f99-a450-658f6de328bf","Type":"ContainerStarted","Data":"5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89"} Nov 01 08:53:09.663108 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:09.662661 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" event={"ID":"1ec208b0-7f97-4f99-a450-658f6de328bf","Type":"ContainerStarted","Data":"021e1cab253f2ebe0f9a1be3cf528677860f680914a25c89cba7b87b06731c0a"} Nov 01 08:53:09.823778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:09.823692 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podStartSLOduration=1.823676173 podStartE2EDuration="1.823676173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:09.822226016 +0000 UTC m=+1212.910562679" watchObservedRunningTime="2026-11-01 08:53:09.823676173 +0000 UTC m=+1212.912012844" Nov 01 08:53:10.463454 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:10.463416 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:10.465322 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:10.465293 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:10.465453 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:10.465345 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:11.463284 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:11.463242 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:11.463761 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:11.463301 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:12.463359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.463322 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:12.463796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.463391 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:12.536922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.536884 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8"] Nov 01 08:53:12.541945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.541919 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.624616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.621805 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8"] Nov 01 08:53:12.658506 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.658457 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.659920 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.659886 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.659927 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.659965 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.659983 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.660006 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660080 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.660077 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2zt\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660341 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.660116 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.660341 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.660134 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761604 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761500 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761604 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761542 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761604 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761566 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761628 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2zt\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761665 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761683 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.761938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761784 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761962 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.761996 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.762127 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762309 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.762218 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762309 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.762290 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762413 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.762370 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.762673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.762648 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.764208 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.764183 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.764443 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.764424 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.819687 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.819654 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.819875 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.819800 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2zt\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:12.857304 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:12.857270 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:13.327123 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.327083 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8"] Nov 01 08:53:13.333526 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:53:13.333499 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d956d1_cc92_4492_98a0_6e485dc9fd0d.slice/crio-977b2c37299fde268a667b12da7ab9d48521ad801290a0ba4dde89768715c045 WatchSource:0}: Error finding container 977b2c37299fde268a667b12da7ab9d48521ad801290a0ba4dde89768715c045: Status 404 returned error can't find the container with id 977b2c37299fde268a667b12da7ab9d48521ad801290a0ba4dde89768715c045 Nov 01 08:53:13.336170 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.336138 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:13.336252 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.336207 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:13.336252 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.336239 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:13.463120 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.463083 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:13.463305 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.463140 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:13.721544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.721505 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" event={"ID":"c0d956d1-cc92-4492-98a0-6e485dc9fd0d","Type":"ContainerStarted","Data":"b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2"} Nov 01 08:53:13.722137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.721567 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" event={"ID":"c0d956d1-cc92-4492-98a0-6e485dc9fd0d","Type":"ContainerStarted","Data":"977b2c37299fde268a667b12da7ab9d48521ad801290a0ba4dde89768715c045"} Nov 01 08:53:13.838446 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.838393 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podStartSLOduration=1.838376198 podStartE2EDuration="1.838376198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:13.835044045 +0000 UTC m=+1216.923380716" watchObservedRunningTime="2026-11-01 08:53:13.838376198 +0000 UTC m=+1216.926712871" Nov 01 08:53:13.857775 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:13.857736 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:14.021806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:14.020827 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:53:14.462989 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:14.462949 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:14.463193 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:14.463015 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:14.866749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:14.860891 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:14.866749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:14.860953 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:15.464748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:15.464368 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:15.464748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:15.464450 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:15.769807 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:53:15.768991 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:53:15.916747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:15.904426 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:15.916747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:15.904532 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:16.463255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:16.463210 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:16.463576 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:16.463549 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:16.867751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:16.867052 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:16.867751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:16.867161 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:16.962401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:16.961939 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:17.464332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.464292 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 08:53:17.464685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.464634 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 08:53:17.806938 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.806832 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:53:17.807226 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.807193 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:17.861291 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.861255 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:17.861590 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.861561 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:17.882776 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:17.881909 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:53:18.102847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.076853 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:18.102847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.077389 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:18.102847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.077831 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:53:18.808749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.764342 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:18.897333 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.896023 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:18.912749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.908779 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:18.944746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.939157 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:18.944746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:18.939234 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:19.077649 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.077557 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 08:53:19.525744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.524907 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:53:19.525744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.524991 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:19.776419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.776308 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:19.778632 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.778596 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:19.780245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.780211 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:19.781236 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.781216 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:19.782940 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.782920 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 08:53:19.891423 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.891373 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:19.892796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.892756 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:19.898162 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:19.898119 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:53:20.125746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:20.122784 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:20.509850 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:20.509216 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:20.778260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:20.778160 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:20.881065 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:20.881022 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:53:20.999006 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:20.998965 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:20.999314 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:20.999281 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:21.857781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:21.857698 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:21.858257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:21.857801 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:22.858375 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:22.858337 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 08:53:22.858817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:22.858405 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 08:53:23.892191 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:23.892152 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:23.893706 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:23.893672 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:23.894534 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:23.894513 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 08:53:32.125821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:32.124819 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:32.935745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:32.934828 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 01 08:53:33.077744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:33.060825 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:33.172747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:33.170889 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:33.172747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:33.171670 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:53:33.939624 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:33.913402 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:34.088756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:34.072247 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:34.136747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:34.103446 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:34.318798 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:34.183982 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:34.966782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:34.958120 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:34.966782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:34.962092 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:35.095741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:35.094572 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:53:35.365278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:35.365220 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:35.958766 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:35.958695 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:35.997912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:35.997872 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:53:35.998200 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:35.998168 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:36.170749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:36.166473 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:53:36.252843 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:36.252708 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:36.388860 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:36.388809 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 08:53:36.977576 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:36.977505 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:37.030069 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:37.029056 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:53:37.182089 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:37.182062 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:45.477976 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:53:45.477936 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:53:50.140282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:50.140246 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:53:52.993803 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:52.969602 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:53.118751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.094989 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:53:53.143394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.100183 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:53.417420 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.413564 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:53.553827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.550442 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz"] Nov 01 08:53:53.585304 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.585271 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.641345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.641310 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz"] Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.678877 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.678984 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679138 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679169 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thqm\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679208 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679255 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679285 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679363 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.685813 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.679417 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786081 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786249 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786286 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thqm\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786319 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786364 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786391 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786471 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786525 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.786604 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.830749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.808490 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.832209 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.832172 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.859747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.833784 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.859747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.834115 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:53.859747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:53.835099 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:54.091829 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.080488 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:54.091829 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.089692 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thqm\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:54.144871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.144816 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:54.173554 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.173518 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:54.229769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.224901 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:53:54.232841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.230125 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:54.232841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.230572 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:54.232841 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.231947 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo\") pod \"e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:54.542818 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.529044 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:54.948255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.947843 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:53:54.948255 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:54.947917 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:55.159946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:55.159878 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:53:55.227038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:55.226879 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:56.177517 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.176319 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:53:56.275622 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.273863 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:53:56.432862 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.432764 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz"] Nov 01 08:53:56.540358 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:53:56.540314 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63531f67_6d85_41d6_b72f_f067225aca11.slice/crio-46b999b3c0b0758987d54032b3c4785881d7d4d5eead9120cd3a3468af079779 WatchSource:0}: Error finding container 46b999b3c0b0758987d54032b3c4785881d7d4d5eead9120cd3a3468af079779: Status 404 returned error can't find the container with id 46b999b3c0b0758987d54032b3c4785881d7d4d5eead9120cd3a3468af079779 Nov 01 08:53:56.569181 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.551985 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:53:56.572580 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.572532 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:56.573038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.572906 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:56.573038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:56.572988 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:53:57.006366 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:57.006336 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:53:57.108760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:57.105574 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" event={"ID":"63531f67-6d85-41d6-b72f-f067225aca11","Type":"ContainerStarted","Data":"46b999b3c0b0758987d54032b3c4785881d7d4d5eead9120cd3a3468af079779"} Nov 01 08:53:58.088519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:58.087739 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" event={"ID":"63531f67-6d85-41d6-b72f-f067225aca11","Type":"ContainerStarted","Data":"bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4"} Nov 01 08:53:58.301533 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:58.296828 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podStartSLOduration=5.296804758 podStartE2EDuration="5.296804758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:53:58.250024875 +0000 UTC m=+1261.338361548" watchObservedRunningTime="2026-11-01 08:53:58.296804758 +0000 UTC m=+1261.385141431" Nov 01 08:53:58.538958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:58.538922 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:53:58.540731 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:58.540690 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 08:53:58.540855 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:58.540771 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 08:53:59.539491 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:59.539450 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 08:53:59.539984 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:53:59.539560 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 08:54:00.539290 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:00.539251 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 08:54:00.539479 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:00.539323 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 08:54:01.539910 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:01.539866 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 08:54:01.540429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:01.539958 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 08:54:02.539278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:02.539245 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 08:54:02.539466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:02.539296 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 08:54:03.539614 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:03.539569 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 08:54:03.540068 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:03.539638 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 08:54:05.122013 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:05.121914 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:54:05.122596 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:05.122562 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:54:05.123781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:05.123761 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 08:54:10.177964 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:10.177892 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:10.238781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:10.236814 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:10.393746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:10.392817 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:11.230224 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.227055 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:11.478743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.409346 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:11.478743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.431048 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 01 08:54:11.478743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.431419 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:54:11.966282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.960576 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:54:11.966282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.960642 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:54:11.966282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.960681 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:11.966282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.960711 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:11.966282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.960761 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:54:11.966282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:11.965627 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:54:15.482539 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:54:15.481695 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:54:22.474333 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.474299 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2"] Nov 01 08:54:22.478963 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.478937 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.533855 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.532627 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d31acd45-openshift-default-dockercfg-p6mlc\"" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.552879 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.552951 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553021 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553134 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2fv\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553208 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553242 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553277 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553322 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.555744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.553385 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.657649 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.637783 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2"] Nov 01 08:54:22.684640 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.666105 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685058 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2fv\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685174 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685215 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685254 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685307 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685386 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685481 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.685531 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.686753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.686058 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.689201 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.687607 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.689201 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.687964 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.689201 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.688571 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.708779 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.691762 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.725569 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.725451 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.792748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.765444 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.870904 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.870814 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:22.898550 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:22.898471 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2fv\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv\") pod \"e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:23.129405 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:23.129360 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:23.566739 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:23.566638 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2"] Nov 01 08:54:23.573448 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:54:23.573417 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25050d6_773f_4e20_81ef_7f8f0ab796ed.slice/crio-55bfb261f0fda3d03f937fe9fd97703b1f5d5abb4df5d5b8e6457a03a99ba7cb WatchSource:0}: Error finding container 55bfb261f0fda3d03f937fe9fd97703b1f5d5abb4df5d5b8e6457a03a99ba7cb: Status 404 returned error can't find the container with id 55bfb261f0fda3d03f937fe9fd97703b1f5d5abb4df5d5b8e6457a03a99ba7cb Nov 01 08:54:23.576196 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:23.576160 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:54:23.576300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:23.576237 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:54:23.576300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:23.576271 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:54:24.062868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:24.062825 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" event={"ID":"d25050d6-773f-4e20-81ef-7f8f0ab796ed","Type":"ContainerStarted","Data":"833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969"} Nov 01 08:54:24.063065 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:24.062935 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" event={"ID":"d25050d6-773f-4e20-81ef-7f8f0ab796ed","Type":"ContainerStarted","Data":"55bfb261f0fda3d03f937fe9fd97703b1f5d5abb4df5d5b8e6457a03a99ba7cb"} Nov 01 08:54:24.130103 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:24.130066 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:24.135875 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:24.135844 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:24.223560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:24.223506 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podStartSLOduration=2.223489259 podStartE2EDuration="2.223489259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:54:24.222204204 +0000 UTC m=+1287.310540876" watchObservedRunningTime="2026-11-01 08:54:24.223489259 +0000 UTC m=+1287.311825930" Nov 01 08:54:25.068772 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:25.068737 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:25.070068 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:25.070043 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:54:27.481004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:27.480962 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:54:29.975916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:29.975851 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:30.172624 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:30.172573 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:30.234951 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:30.234834 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:30.585922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:30.585823 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:54:30.586221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:30.586180 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:31.350746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.185130 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:31.399374 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.185831 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:31.472871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.257004 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:31.473220 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.257071 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:31.474679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.450285 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:54:31.474999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.474969 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:54:31.476009 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.475989 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:31.786432 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.786330 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:54:31.786751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.786695 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:31.892751 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:31.890845 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:32.385999 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:32.385953 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:32.497748 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:32.489532 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 08:54:32.821347 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:32.821229 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:54:32.821621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:32.821586 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:32.984749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:32.980875 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:33.020749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:33.019017 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:33.402741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:33.391316 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:33.520798 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:33.518843 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:34.004747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:33.995706 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:54:34.004747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:33.996636 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:34.601747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:34.496091 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:34.601747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:34.533848 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 08:54:34.648910 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:34.648862 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lnhsp" podUID="0c2579b5-5533-45d8-a0b6-c4272b055f20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:35.001836 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:34.999095 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:35.365484 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:35.365455 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:54:35.382449 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:35.382417 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:54:35.407553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:35.407522 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:45.478759 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:54:45.478698 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:54:55.409147 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:55.409078 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 08:54:55.466749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:55.438459 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:54:56.498747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:56.421285 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:54:56.498747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:56.447629 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:56.607747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:56.603612 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:54:56.671989 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:56.671959 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:54:56.672331 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:56.672312 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:54:57.425996 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:57.425951 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:54:57.474663 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:57.474613 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:57.781934 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:57.647626 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:57.804238 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:57.798899 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:54:58.098815 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:58.090922 5710 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Nov 01 08:54:58.502807 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:58.500383 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:58.527745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:58.525911 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:54:58.861005 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:58.837036 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:54:58.952744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:58.943980 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 08:54:59.523152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:59.523107 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:54:59.567533 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:59.567491 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5"] Nov 01 08:54:59.785827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:59.778175 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5"] Nov 01 08:54:59.785827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:59.778301 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:54:59.866921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:59.866860 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:00.018743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:54:59.908194 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d31acd45-serving-cert\"" Nov 01 08:55:00.135746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.120680 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.135746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.120869 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.135746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.121081 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.135746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.121139 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwwp\" (UniqueName: \"kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.270759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.262200 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.270759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.262295 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwwp\" (UniqueName: \"kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.270759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.262498 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.270759 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.262600 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.398795 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.398065 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.415810 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.410421 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.653858 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.653777 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.680060 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.676166 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 01 08:55:00.727805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.674711 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwwp\" (UniqueName: \"kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:00.738885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.738834 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:00.782593 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:00.782556 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:01.062745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:01.050688 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 08:55:01.180763 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:01.148482 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:01.415137 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:01.415102 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:55:02.177313 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:02.176860 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5"] Nov 01 08:55:02.241864 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:02.241839 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 08:55:02.261909 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:02.261854 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:55:02.262318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:02.262209 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:02.796058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:02.796012 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" event={"ID":"3ffff0af-66fc-4b11-9853-60ab0709a496","Type":"ContainerStarted","Data":"6b66e3178b8c4a0d84efc3aeba5bdff1c3b6964bbe57fec2fa968fe19e25e138"} Nov 01 08:55:03.880799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:03.872089 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" event={"ID":"3ffff0af-66fc-4b11-9853-60ab0709a496","Type":"ContainerStarted","Data":"e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4"} Nov 01 08:55:03.887596 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:03.887066 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:04.008116 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.008015 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" podStartSLOduration=3.969394218 podStartE2EDuration="5.007990158s" totalImagesPullingTime="1.03859594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:04.00311758 +0000 UTC m=+1327.091454256" watchObservedRunningTime="2026-11-01 08:55:04.007990158 +0000 UTC m=+1327.096326832" Nov 01 08:55:04.365430 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.365387 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z"] Nov 01 08:55:04.370394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.370364 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.424151 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.424115 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z"] Nov 01 08:55:04.489687 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.489651 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.489954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.489695 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.489954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.489806 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.489954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.489926 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdt66\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.542586 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.542550 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j"] Nov 01 08:55:04.547142 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.547112 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591067 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591034 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84tm\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591279 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591141 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591354 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591266 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591354 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591318 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.591509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591476 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591576 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591522 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591630 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591588 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.591683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591632 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.591683 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591659 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.591845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.591786 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdt66\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.592036 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.592001 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.592131 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.592061 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.592131 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.592104 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.592246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.592188 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.592517 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.592470 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.595125 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.595102 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.605523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.599108 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-93665b-openshift-default-dockercfg-2v6ld\"" Nov 01 08:55:04.605523 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.605704 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j"] Nov 01 08:55:04.658567 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.658489 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdt66\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66\") pod \"payload-processing-external-model-b6fcd7955-p2n7z\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.687905 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.687866 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:04.693328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693288 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84tm\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693487 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693379 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693487 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693435 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693620 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693575 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693620 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693599 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693664 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693744 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693770 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.693804 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693796 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.694040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.693947 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.694411 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.694387 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.694547 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.694527 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.694672 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.694654 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.694893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.694872 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.696420 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.696400 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.697553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.697528 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.759363 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.758988 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.761531 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.761485 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84tm\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm\") pod \"e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:04.945467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:04.941072 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:05.471341 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:05.469883 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z"] Nov 01 08:55:05.648860 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:05.648812 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j"] Nov 01 08:55:05.654628 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:55:05.654582 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce47b1f_78db_4688_a11a_2ed9daa86803.slice/crio-5e0665813637dd1cc4a5cc3afd14c71ff1aadc376d976b5a7e698f6b78decc67 WatchSource:0}: Error finding container 5e0665813637dd1cc4a5cc3afd14c71ff1aadc376d976b5a7e698f6b78decc67: Status 404 returned error can't find the container with id 5e0665813637dd1cc4a5cc3afd14c71ff1aadc376d976b5a7e698f6b78decc67 Nov 01 08:55:05.658166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:05.658125 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:05.658291 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:05.658222 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:05.658291 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:05.658265 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:06.050929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.050893 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" event={"ID":"0ce47b1f-78db-4688-a11a-2ed9daa86803","Type":"ContainerStarted","Data":"61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063"} Nov 01 08:55:06.051419 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.050944 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" event={"ID":"0ce47b1f-78db-4688-a11a-2ed9daa86803","Type":"ContainerStarted","Data":"5e0665813637dd1cc4a5cc3afd14c71ff1aadc376d976b5a7e698f6b78decc67"} Nov 01 08:55:06.052929 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.052898 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" event={"ID":"6ca8b677-a478-4047-bd51-5f7ca7a0819e","Type":"ContainerStarted","Data":"3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3"} Nov 01 08:55:06.053054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.052950 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" event={"ID":"6ca8b677-a478-4047-bd51-5f7ca7a0819e","Type":"ContainerStarted","Data":"074dc4fcc865b8feb2b2031fc7bd58c5f6b5ed78fa57c23b2944fbd3acdce0fb"} Nov 01 08:55:06.053054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.052981 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:06.165895 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.165836 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podStartSLOduration=2.165819077 podStartE2EDuration="2.165819077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:06.163543967 +0000 UTC m=+1329.251880650" watchObservedRunningTime="2026-11-01 08:55:06.165819077 +0000 UTC m=+1329.254155748" Nov 01 08:55:06.274907 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.274406 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" podStartSLOduration=2.274380004 podStartE2EDuration="2.274380004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:06.272680014 +0000 UTC m=+1329.361016683" watchObservedRunningTime="2026-11-01 08:55:06.274380004 +0000 UTC m=+1329.362716677" Nov 01 08:55:06.944271 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.944230 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:06.951565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:06.951531 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:07.060778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:07.060744 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:07.061664 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:07.061645 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:55:10.048206 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:10.048176 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:55:12.615739 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.590785 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl"] Nov 01 08:55:12.633741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.628328 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:12.643560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.643524 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl"] Nov 01 08:55:12.733402 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.733368 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-93665b-dockercfg-7twg2\"" Nov 01 08:55:12.733823 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.733791 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-93665b\"" Nov 01 08:55:12.831260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.785027 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj"] Nov 01 08:55:12.852744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.844223 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8r28\" (UniqueName: \"kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28\") pod \"payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:12.857108 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.857074 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:12.939232 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.939125 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj"] Nov 01 08:55:12.940871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.939590 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:12.975004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.968159 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8r28\" (UniqueName: \"kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28\") pod \"payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:12.975004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.968265 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:12.975004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.968457 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93665b-5d94689985-srgzj\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:12.975004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.968606 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjdn\" (UniqueName: \"kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn\") pod \"payload-processing-e2e-api-a-93665b-5d94689985-srgzj\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:13.003521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:12.985203 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:13.032790 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.031392 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8r28\" (UniqueName: \"kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28\") pod \"payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:13.071436 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.071394 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93665b-5d94689985-srgzj\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:13.071637 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.071535 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjdn\" (UniqueName: \"kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn\") pod \"payload-processing-e2e-api-a-93665b-5d94689985-srgzj\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:13.072373 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.072350 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93665b-5d94689985-srgzj\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:13.129372 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.129333 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjdn\" (UniqueName: \"kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn\") pod \"payload-processing-e2e-api-a-93665b-5d94689985-srgzj\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:13.272318 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.272221 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:13.323638 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.323606 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:13.652893 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.652839 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl"] Nov 01 08:55:13.659796 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:55:13.659753 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18cf697_e205_410b_bb97_bb56b8b6ff81.slice/crio-e11479bfc506a62ed315e2a8fed3ec556480a7c0c671a263ebd3b503842c5300 WatchSource:0}: Error finding container e11479bfc506a62ed315e2a8fed3ec556480a7c0c671a263ebd3b503842c5300: Status 404 returned error can't find the container with id e11479bfc506a62ed315e2a8fed3ec556480a7c0c671a263ebd3b503842c5300 Nov 01 08:55:13.760757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:13.760704 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj"] Nov 01 08:55:13.767342 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:55:13.767314 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0016e9_e03b_433a_a75e_c3f546a53c77.slice/crio-21d3de00f792c040ecbe838f167980c03ead724f56b0cb6d28633bf2bca095fc WatchSource:0}: Error finding container 21d3de00f792c040ecbe838f167980c03ead724f56b0cb6d28633bf2bca095fc: Status 404 returned error can't find the container with id 21d3de00f792c040ecbe838f167980c03ead724f56b0cb6d28633bf2bca095fc Nov 01 08:55:14.109401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.109364 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" event={"ID":"ce0016e9-e03b-433a-a75e-c3f546a53c77","Type":"ContainerStarted","Data":"77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213"} Nov 01 08:55:14.109633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.109427 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:14.109633 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.109452 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" event={"ID":"ce0016e9-e03b-433a-a75e-c3f546a53c77","Type":"ContainerStarted","Data":"21d3de00f792c040ecbe838f167980c03ead724f56b0cb6d28633bf2bca095fc"} Nov 01 08:55:14.111039 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.111008 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" event={"ID":"d18cf697-e205-410b-bb97-bb56b8b6ff81","Type":"ContainerStarted","Data":"93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e"} Nov 01 08:55:14.111164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.111052 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" event={"ID":"d18cf697-e205-410b-bb97-bb56b8b6ff81","Type":"ContainerStarted","Data":"e11479bfc506a62ed315e2a8fed3ec556480a7c0c671a263ebd3b503842c5300"} Nov 01 08:55:14.111240 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.111153 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:14.218411 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.218356 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" podStartSLOduration=2.218341134 podStartE2EDuration="2.218341134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:14.217672496 +0000 UTC m=+1337.306009166" watchObservedRunningTime="2026-11-01 08:55:14.218341134 +0000 UTC m=+1337.306677802" Nov 01 08:55:14.346753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:14.342238 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" podStartSLOduration=2.342215068 podStartE2EDuration="2.342215068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:14.33912393 +0000 UTC m=+1337.427460615" watchObservedRunningTime="2026-11-01 08:55:14.342215068 +0000 UTC m=+1337.430551745" Nov 01 08:55:15.479928 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:55:15.479884 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:55:17.064156 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:17.064118 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:55:17.529265 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:17.529222 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z"] Nov 01 08:55:17.553372 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:17.552896 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 08:55:17.582871 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:17.580277 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn"] Nov 01 08:55:22.530044 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.529999 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerName="payload-processing" containerID="cri-o://3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3" gracePeriod=25 Nov 01 08:55:22.581345 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.581285 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" podUID="0725d742-71e3-46ba-93b7-5f31c1671081" containerName="payload-processing" containerID="cri-o://7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013" gracePeriod=25 Nov 01 08:55:22.895753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.895419 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:22.900056 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.900025 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:55:22.961175 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961133 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66\") pod \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " Nov 01 08:55:22.961367 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961200 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials\") pod \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " Nov 01 08:55:22.961367 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961281 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume\") pod \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " Nov 01 08:55:22.961367 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961321 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4\") pod \"0725d742-71e3-46ba-93b7-5f31c1671081\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " Nov 01 08:55:22.961470 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961371 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume\") pod \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\" (UID: \"6ca8b677-a478-4047-bd51-5f7ca7a0819e\") " Nov 01 08:55:22.961470 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961455 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials\") pod \"0725d742-71e3-46ba-93b7-5f31c1671081\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " Nov 01 08:55:22.961562 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961526 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume\") pod \"0725d742-71e3-46ba-93b7-5f31c1671081\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " Nov 01 08:55:22.961651 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961622 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume\") pod \"0725d742-71e3-46ba-93b7-5f31c1671081\" (UID: \"0725d742-71e3-46ba-93b7-5f31c1671081\") " Nov 01 08:55:22.962258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961706 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume" pod "6ca8b677-a478-4047-bd51-5f7ca7a0819e" (UID: "6ca8b677-a478-4047-bd51-5f7ca7a0819e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:55:22.962258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.961841 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume" pod "6ca8b677-a478-4047-bd51-5f7ca7a0819e" (UID: "6ca8b677-a478-4047-bd51-5f7ca7a0819e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:55:22.962442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.962271 5710 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-routing-overlay-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:22.962442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.962290 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca8b677-a478-4047-bd51-5f7ca7a0819e-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:22.962442 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.962421 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume" pod "0725d742-71e3-46ba-93b7-5f31c1671081" (UID: "0725d742-71e3-46ba-93b7-5f31c1671081"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:55:22.962538 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.962139 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume" pod "0725d742-71e3-46ba-93b7-5f31c1671081" (UID: "0725d742-71e3-46ba-93b7-5f31c1671081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:55:22.963737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.963700 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66" pod "6ca8b677-a478-4047-bd51-5f7ca7a0819e" (UID: "6ca8b677-a478-4047-bd51-5f7ca7a0819e"). InnerVolumeSpecName "kube-api-access-tdt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:55:22.963866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.963767 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials" pod "0725d742-71e3-46ba-93b7-5f31c1671081" (UID: "0725d742-71e3-46ba-93b7-5f31c1671081"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:55:22.964150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.964133 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4" pod "0725d742-71e3-46ba-93b7-5f31c1671081" (UID: "0725d742-71e3-46ba-93b7-5f31c1671081"). InnerVolumeSpecName "kube-api-access-ls9z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:55:22.964193 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:22.964144 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials" pod "6ca8b677-a478-4047-bd51-5f7ca7a0819e" (UID: "6ca8b677-a478-4047-bd51-5f7ca7a0819e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:55:23.063742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.063644 5710 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-routing-overlay-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:23.063742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.063673 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdt66\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-kube-api-access-tdt66\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:23.063742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.063683 5710 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ca8b677-a478-4047-bd51-5f7ca7a0819e-provider-credentials\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:23.063742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.063693 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9z4\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-kube-api-access-ls9z4\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:23.063742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.063702 5710 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0725d742-71e3-46ba-93b7-5f31c1671081-provider-credentials\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:23.063742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.063712 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0725d742-71e3-46ba-93b7-5f31c1671081-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:55:23.248670 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.248635 5710 generic.go:356] "Generic (PLEG): container finished" podID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerID="3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3" exitCode=0 Nov 01 08:55:23.248901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.248699 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" Nov 01 08:55:23.248901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.248752 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" event={"ID":"6ca8b677-a478-4047-bd51-5f7ca7a0819e","Type":"ContainerDied","Data":"3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3"} Nov 01 08:55:23.248901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.248809 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z" event={"ID":"6ca8b677-a478-4047-bd51-5f7ca7a0819e","Type":"ContainerDied","Data":"074dc4fcc865b8feb2b2031fc7bd58c5f6b5ed78fa57c23b2944fbd3acdce0fb"} Nov 01 08:55:23.248901 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.248831 5710 scope.go:122] "RemoveContainer" containerID="3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3" Nov 01 08:55:23.250437 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.250413 5710 generic.go:356] "Generic (PLEG): container finished" podID="0725d742-71e3-46ba-93b7-5f31c1671081" containerID="7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013" exitCode=0 Nov 01 08:55:23.250737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.250477 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" event={"ID":"0725d742-71e3-46ba-93b7-5f31c1671081","Type":"ContainerDied","Data":"7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013"} Nov 01 08:55:23.250737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.250511 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" event={"ID":"0725d742-71e3-46ba-93b7-5f31c1671081","Type":"ContainerDied","Data":"4a27f60e16701e315ae3558bda1b001a0c3d086d45ef66c82db559ead363a551"} Nov 01 08:55:23.250737 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.250480 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn" Nov 01 08:55:23.259526 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.259503 5710 scope.go:122] "RemoveContainer" containerID="3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3" Nov 01 08:55:23.259874 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:55:23.259851 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3\": container with ID starting with 3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3 not found: ID does not exist" containerID="3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3" Nov 01 08:55:23.259935 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.259889 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3"} err="failed to get container status \"3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3\": rpc error: code = NotFound desc = could not find container \"3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3\": container with ID starting with 3b423f25cda8365bd10e7cdcd2636dd83b364ee10bb42acb9e6175c5001649c3 not found: ID does not exist" Nov 01 08:55:23.259935 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.259910 5710 scope.go:122] "RemoveContainer" containerID="7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013" Nov 01 08:55:23.272050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.272024 5710 scope.go:122] "RemoveContainer" containerID="7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013" Nov 01 08:55:23.272420 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:55:23.272395 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013\": container with ID starting with 7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013 not found: ID does not exist" containerID="7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013" Nov 01 08:55:23.272493 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.272427 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013"} err="failed to get container status \"7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013\": rpc error: code = NotFound desc = could not find container \"7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013\": container with ID starting with 7282d6cb4308d64839702b548ef7783957a81d1a7625ef367c5001ff93157013 not found: ID does not exist" Nov 01 08:55:23.404116 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.404083 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn"] Nov 01 08:55:23.407329 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.407302 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c69ddc68f-xd8xn"] Nov 01 08:55:23.508362 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.508330 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z"] Nov 01 08:55:23.511278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:23.511255 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-b6fcd7955-p2n7z"] Nov 01 08:55:25.060011 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:25.059981 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0725d742-71e3-46ba-93b7-5f31c1671081" path="/var/lib/kubelet/pods/0725d742-71e3-46ba-93b7-5f31c1671081/volumes" Nov 01 08:55:25.060447 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:25.060433 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" path="/var/lib/kubelet/pods/6ca8b677-a478-4047-bd51-5f7ca7a0819e/volumes" Nov 01 08:55:25.121467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:25.121437 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:55:25.121757 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:25.121739 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:55:26.817741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:26.817296 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:55:30.992896 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:30.992834 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:31.489750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:31.488883 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:31.999766 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:31.998843 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 01 08:55:31.999766 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:31.999560 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:55:32.349812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:32.337841 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-xk6c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:55:32.349812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:32.337939 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:55:32.429491 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:32.429455 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 08:55:38.487343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:38.487246 5710 generic.go:356] "Generic (PLEG): container finished" podID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerID="833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969" exitCode=0 Nov 01 08:55:38.487343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:38.487321 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" event={"ID":"d25050d6-773f-4e20-81ef-7f8f0ab796ed","Type":"ContainerDied","Data":"833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969"} Nov 01 08:55:38.487921 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:38.487836 5710 scope.go:122] "RemoveContainer" containerID="833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969" Nov 01 08:55:38.489625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:38.489592 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:38.489760 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:38.489682 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:38.489814 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:38.489760 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:39.546797 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:39.542149 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" event={"ID":"d25050d6-773f-4e20-81ef-7f8f0ab796ed","Type":"ContainerStarted","Data":"6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36"} Nov 01 08:55:40.069259 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:40.069224 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:55:40.129845 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:40.129796 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:55:40.131166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:40.131137 5710 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 08:55:40.131323 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:40.131222 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 08:55:41.130257 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:41.130227 5710 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 08:55:41.130668 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:41.130274 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 08:55:42.129943 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:42.129907 5710 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 08:55:42.130129 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:42.129965 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 08:55:43.130444 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:43.130401 5710 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 08:55:43.131041 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:43.130469 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 08:55:44.130829 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:44.130795 5710 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 08:55:44.131239 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:44.130848 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 08:55:45.130824 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:45.130786 5710 patch_prober.go:29] interesting pod/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 08:55:45.131263 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:45.130843 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 08:55:45.481557 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:55:45.481459 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:55:46.134912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:46.134880 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:55:46.136185 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:46.136164 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 08:55:54.064479 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.064435 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl"] Nov 01 08:55:54.065399 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.065376 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerName="payload-processing" Nov 01 08:55:54.065512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.065401 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerName="payload-processing" Nov 01 08:55:54.065512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.065442 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0725d742-71e3-46ba-93b7-5f31c1671081" containerName="payload-processing" Nov 01 08:55:54.065512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.065450 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0725d742-71e3-46ba-93b7-5f31c1671081" containerName="payload-processing" Nov 01 08:55:54.065673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.065608 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0725d742-71e3-46ba-93b7-5f31c1671081" containerName="payload-processing" Nov 01 08:55:54.065673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.065636 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca8b677-a478-4047-bd51-5f7ca7a0819e" containerName="payload-processing" Nov 01 08:55:54.069758 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.069730 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.120569 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.120507 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5fad29-openshift-default-dockercfg-gcwfv\"" Nov 01 08:55:54.324792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.322859 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.324792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.322916 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.324792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.322986 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.324792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.323032 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.324792 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.323084 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.333305 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.333266 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.333561 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.333538 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.333769 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.333752 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9w5c\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.333956 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.333941 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.362933 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.362895 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl"] Nov 01 08:55:54.443761 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.443695 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.444187 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.444154 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.444348 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.444335 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.444490 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.444474 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.444588 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.444576 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.444707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.444690 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.445000 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.444980 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.445117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.445104 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.493919 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.493878 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.449712 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9w5c\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.580016 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9w5c\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.494399 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.494629 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.495421 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.501264 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.501485 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.579404 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.598806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.495204 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data\") pod \"e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:54.703594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:54.703560 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:55.142955 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.142914 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl"] Nov 01 08:55:55.187238 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:55:55.187206 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ea95b4_423a_4c19_8437_45dc72aa2ba5.slice/crio-f90dd2bb04eb830845d45d4b7170e3935303ea2322c36da5cfd455e4b2477f50 WatchSource:0}: Error finding container f90dd2bb04eb830845d45d4b7170e3935303ea2322c36da5cfd455e4b2477f50: Status 404 returned error can't find the container with id f90dd2bb04eb830845d45d4b7170e3935303ea2322c36da5cfd455e4b2477f50 Nov 01 08:55:55.189903 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.189867 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:55.190023 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.189937 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:55.190023 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.189972 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:55:55.713383 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.713344 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" event={"ID":"34ea95b4-423a-4c19-8437-45dc72aa2ba5","Type":"ContainerStarted","Data":"d8c96d870070fecfda7fc3fe5223ab40ee22a1f0eababd0821a61fac5c0c6195"} Nov 01 08:55:55.713575 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.713429 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" event={"ID":"34ea95b4-423a-4c19-8437-45dc72aa2ba5","Type":"ContainerStarted","Data":"f90dd2bb04eb830845d45d4b7170e3935303ea2322c36da5cfd455e4b2477f50"} Nov 01 08:55:55.828618 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:55.828550 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podStartSLOduration=1.828533716 podStartE2EDuration="1.828533716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:55:55.825973854 +0000 UTC m=+1378.914310525" watchObservedRunningTime="2026-11-01 08:55:55.828533716 +0000 UTC m=+1378.916870387" Nov 01 08:55:56.705116 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:56.705083 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:56.710418 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:56.710391 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:56.720039 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:56.720009 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:56.721023 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:56.720996 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:55:58.947058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:55:58.947021 5710 scope.go:122] "RemoveContainer" containerID="6d37421240030f24acc4ddf83eba56e32785e67aab0c05f1723ffbd4a29b10a7" Nov 01 08:56:02.340113 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.340069 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62"] Nov 01 08:56:02.346324 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.346287 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.368198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.368158 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.368710 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.368682 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbj6w\" (UniqueName: \"kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w\") pod \"payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.424772 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.419321 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62"] Nov 01 08:56:02.449041 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.448996 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5fad29-dockercfg-kqzlw\"" Nov 01 08:56:02.449371 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.449352 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5fad29\"" Nov 01 08:56:02.475762 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.474678 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbj6w\" (UniqueName: \"kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w\") pod \"payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.475762 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.475075 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.489749 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.476185 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.630145 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.630053 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7"] Nov 01 08:56:02.636053 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.636023 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7"] Nov 01 08:56:02.636357 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.636327 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:02.641437 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.639408 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbj6w\" (UniqueName: \"kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w\") pod \"payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.685398 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.685353 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:02.724298 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.724259 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:02.724951 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.724925 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4zd\" (UniqueName: \"kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd\") pod \"payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:02.826590 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.826551 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:02.826846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.826815 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4zd\" (UniqueName: \"kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd\") pod \"payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:02.827632 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.827600 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:02.884684 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:02.884610 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4zd\" (UniqueName: \"kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd\") pod \"payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:03.008765 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.007808 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:03.151081 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.147131 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62"] Nov 01 08:56:03.726227 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.714569 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7"] Nov 01 08:56:03.833987 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.833944 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" event={"ID":"1645573a-e4c5-42f5-8671-9c68931b532c","Type":"ContainerStarted","Data":"6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5"} Nov 01 08:56:03.835332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.835096 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:03.835332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.835166 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" event={"ID":"1645573a-e4c5-42f5-8671-9c68931b532c","Type":"ContainerStarted","Data":"6e8976eb38479f7329c6aa4717dc0177f8dee66c60672ef944e60a6560035cc8"} Nov 01 08:56:03.837329 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.837298 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" event={"ID":"53a7f922-e7f1-4aa3-9858-7acddab9c950","Type":"ContainerStarted","Data":"52fc04ec75ceab26f00bb32c4dc1471e471b62d0c26c4c417e8881c46fcf0e2c"} Nov 01 08:56:03.948182 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:03.948122 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" podStartSLOduration=1.9481065910000002 podStartE2EDuration="1.948106591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:03.944199662 +0000 UTC m=+1387.032536335" watchObservedRunningTime="2026-11-01 08:56:03.948106591 +0000 UTC m=+1387.036443262" Nov 01 08:56:04.850506 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:04.850460 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" event={"ID":"53a7f922-e7f1-4aa3-9858-7acddab9c950","Type":"ContainerStarted","Data":"ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8"} Nov 01 08:56:04.850961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:04.850544 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:04.961615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:04.961557 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" podStartSLOduration=2.9615368650000002 podStartE2EDuration="2.961536865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:04.958762229 +0000 UTC m=+1388.047098896" watchObservedRunningTime="2026-11-01 08:56:04.961536865 +0000 UTC m=+1388.049873536" Nov 01 08:56:14.853685 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.853652 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:56:14.951301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.951270 5710 generic.go:356] "Generic (PLEG): container finished" podID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerID="61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063" exitCode=0 Nov 01 08:56:14.951476 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.951344 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" event={"ID":"0ce47b1f-78db-4688-a11a-2ed9daa86803","Type":"ContainerDied","Data":"61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063"} Nov 01 08:56:14.951852 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.951836 5710 scope.go:122] "RemoveContainer" containerID="61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063" Nov 01 08:56:14.953422 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.953394 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:56:14.953491 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.953464 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:56:14.953526 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:14.953496 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:56:15.496744 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:56:15.493692 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:56:15.860612 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:15.860584 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:56:15.961353 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:15.961315 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" event={"ID":"0ce47b1f-78db-4688-a11a-2ed9daa86803","Type":"ContainerStarted","Data":"d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e"} Nov 01 08:56:16.943837 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:16.943792 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:56:16.944247 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:16.944055 5710 patch_prober.go:29] interesting pod/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 08:56:16.944247 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:16.944118 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 08:56:17.944258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:17.944215 5710 patch_prober.go:29] interesting pod/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 08:56:17.944677 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:17.944288 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 08:56:18.943899 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:18.943862 5710 patch_prober.go:29] interesting pod/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 08:56:18.944097 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:18.943915 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 08:56:19.948075 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:19.948039 5710 patch_prober.go:29] interesting pod/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 08:56:19.948553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:19.948527 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 08:56:20.943885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:20.943844 5710 patch_prober.go:29] interesting pod/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 08:56:20.944082 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:20.943905 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 08:56:21.947952 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:21.947918 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:56:21.948450 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:21.948431 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:56:21.949729 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:21.949701 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:56:23.172110 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:23.172067 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:56:40.023744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.018780 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k"] Nov 01 08:56:40.026753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.026343 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.065819 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.065778 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k"] Nov 01 08:56:40.092014 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.080586 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8fl\" (UniqueName: \"kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl\") pod \"payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.100401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.100369 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.100940 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.100233 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8701a0\"" Nov 01 08:56:40.101125 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.100191 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8701a0-dockercfg-7brvv\"" Nov 01 08:56:40.205438 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.205400 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8fl\" (UniqueName: \"kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl\") pod \"payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.205745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.205702 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.208112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.208083 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.333616 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.333530 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89"] Nov 01 08:56:40.425930 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.382053 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8fl\" (UniqueName: \"kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl\") pod \"payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.425930 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.399049 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.452362 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.450226 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:40.630958 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.472687 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.654698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.581508 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89"] Nov 01 08:56:40.662763 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.655112 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzgh\" (UniqueName: \"kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh\") pod \"payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.791747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.768333 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.792103 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.772651 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.795745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.792185 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzgh\" (UniqueName: \"kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh\") pod \"payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.852745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.851521 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzgh\" (UniqueName: \"kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh\") pod \"payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:40.927707 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:40.927433 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:41.237687 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.237585 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k"] Nov 01 08:56:41.347744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.347082 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" event={"ID":"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0","Type":"ContainerStarted","Data":"74698ff400c1b7e8d87b6366bbaa2568b1aa3c76c133a8ee5cdfe38ec1cc22d4"} Nov 01 08:56:41.370106 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.369118 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89"] Nov 01 08:56:41.372930 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:56:41.372878 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2ceec5_21ce_4cd2_aec1_edea0872f31d.slice/crio-9834554d1d4ca0e5a5e6c8c5327c5786c00205345048ab7f959ce941d39979f3 WatchSource:0}: Error finding container 9834554d1d4ca0e5a5e6c8c5327c5786c00205345048ab7f959ce941d39979f3: Status 404 returned error can't find the container with id 9834554d1d4ca0e5a5e6c8c5327c5786c00205345048ab7f959ce941d39979f3 Nov 01 08:56:41.425674 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.424487 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l"] Nov 01 08:56:41.455269 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.455228 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.517253 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.517212 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l"] Nov 01 08:56:41.534249 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.534209 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.535486 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.535433 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.535859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.535835 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.536022 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.536003 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.537050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.536230 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.537050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.536286 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.537050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.536575 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.537050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.536634 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmqs\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.537050 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.536712 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639224 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639293 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmqs\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639367 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639417 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639486 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639564 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639604 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639700 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639770 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.639887 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.640185 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.640245 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.640574 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.640847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.640793 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.644385 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.644359 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.645452 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.645358 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.700513 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.700475 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmqs\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.701040 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.701016 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:41.815796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:41.813057 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:42.340669 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.340630 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l"] Nov 01 08:56:42.370779 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.370491 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" event={"ID":"0b2ceec5-21ce-4cd2-aec1-edea0872f31d","Type":"ContainerStarted","Data":"582c8fe4361c498d66f0252f0967be8e3c40ddaaa199e8789f93557b6f8b303b"} Nov 01 08:56:42.370779 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.370596 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" event={"ID":"0b2ceec5-21ce-4cd2-aec1-edea0872f31d","Type":"ContainerStarted","Data":"9834554d1d4ca0e5a5e6c8c5327c5786c00205345048ab7f959ce941d39979f3"} Nov 01 08:56:42.371072 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.371037 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:42.372946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.372915 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" event={"ID":"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0","Type":"ContainerStarted","Data":"802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee"} Nov 01 08:56:42.373202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.373162 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:56:42.377124 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:56:42.377090 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a024d8_ba46_40b6_8490_9475eca60e5d.slice/crio-be588b49d1e17a68510a5a70e425999416333f59c858beb8e991c3d96243159c WatchSource:0}: Error finding container be588b49d1e17a68510a5a70e425999416333f59c858beb8e991c3d96243159c: Status 404 returned error can't find the container with id be588b49d1e17a68510a5a70e425999416333f59c858beb8e991c3d96243159c Nov 01 08:56:42.380583 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.380535 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:56:42.380678 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.380645 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:56:42.380746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.380700 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:56:42.663916 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.660219 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" podStartSLOduration=2.660191506 podStartE2EDuration="2.660191506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:42.535038392 +0000 UTC m=+1425.623375068" watchObservedRunningTime="2026-11-01 08:56:42.660191506 +0000 UTC m=+1425.748528180" Nov 01 08:56:42.716847 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:42.710310 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" podStartSLOduration=3.710290039 podStartE2EDuration="3.710290039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:42.693289656 +0000 UTC m=+1425.781626339" watchObservedRunningTime="2026-11-01 08:56:42.710290039 +0000 UTC m=+1425.798626712" Nov 01 08:56:43.410639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:43.410508 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" event={"ID":"b5a024d8-ba46-40b6-8490-9475eca60e5d","Type":"ContainerStarted","Data":"62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee"} Nov 01 08:56:43.410639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:43.410595 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" event={"ID":"b5a024d8-ba46-40b6-8490-9475eca60e5d","Type":"ContainerStarted","Data":"be588b49d1e17a68510a5a70e425999416333f59c858beb8e991c3d96243159c"} Nov 01 08:56:43.576227 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:43.576149 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podStartSLOduration=2.5761279139999997 podStartE2EDuration="2.576127914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:56:43.570394366 +0000 UTC m=+1426.658731064" watchObservedRunningTime="2026-11-01 08:56:43.576127914 +0000 UTC m=+1426.664464581" Nov 01 08:56:43.819809 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:43.818119 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:43.823747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:43.821903 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:43.823747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:43.821965 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:44.819743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:44.819310 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:44.819743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:44.819406 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:45.697744 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:56:45.695988 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:56:45.845741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:45.817200 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:45.845741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:45.817271 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:46.817282 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:46.817237 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:46.817486 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:46.817304 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:47.817324 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:47.817290 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:47.817753 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:47.817347 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:48.816532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:48.816495 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:48.816746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:48.816554 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:49.817139 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:49.817099 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:49.817521 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:49.817167 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:50.816830 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:50.816785 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 08:56:50.817051 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:50.816858 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 08:56:51.821076 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:51.821047 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:51.821532 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:51.821428 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:51.822106 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:51.822086 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 08:56:53.415997 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:53.415968 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:56:53.416519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:56:53.416048 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:57:07.421165 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:07.421117 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:57:15.478832 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:57:15.478785 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:57:18.913229 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:18.913196 5710 generic.go:356] "Generic (PLEG): container finished" podID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerID="d8c96d870070fecfda7fc3fe5223ab40ee22a1f0eababd0821a61fac5c0c6195" exitCode=0 Nov 01 08:57:18.913817 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:18.913269 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" event={"ID":"34ea95b4-423a-4c19-8437-45dc72aa2ba5","Type":"ContainerDied","Data":"d8c96d870070fecfda7fc3fe5223ab40ee22a1f0eababd0821a61fac5c0c6195"} Nov 01 08:57:18.913940 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:18.913922 5710 scope.go:122] "RemoveContainer" containerID="d8c96d870070fecfda7fc3fe5223ab40ee22a1f0eababd0821a61fac5c0c6195" Nov 01 08:57:18.915569 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:18.915542 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:18.915642 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:18.915615 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:18.915681 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:18.915646 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:19.923559 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:19.923519 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" event={"ID":"34ea95b4-423a-4c19-8437-45dc72aa2ba5","Type":"ContainerStarted","Data":"6361519a82cf1721be71d7f0321b217df3425c8dd2410566ac50403680f6dcec"} Nov 01 08:57:20.705223 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:20.705182 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:57:20.705668 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:20.705574 5710 patch_prober.go:29] interesting pod/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 08:57:20.705954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:20.705904 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 08:57:21.705922 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:21.705882 5710 patch_prober.go:29] interesting pod/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 08:57:21.706447 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:21.705959 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 08:57:22.707742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:22.706333 5710 patch_prober.go:29] interesting pod/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 08:57:22.707742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:22.706401 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 08:57:23.706236 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:23.706194 5710 patch_prober.go:29] interesting pod/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 08:57:23.706458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:23.706271 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 08:57:24.707293 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:24.707252 5710 patch_prober.go:29] interesting pod/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 08:57:24.707876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:24.707839 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 08:57:25.709782 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:25.709749 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:57:25.710263 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:25.710247 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:57:25.711163 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:25.711143 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 08:57:26.768504 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.768471 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh"] Nov 01 08:57:26.773032 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.773011 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.892017 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.891954 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh"] Nov 01 08:57:26.900083 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.900040 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.900394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.900125 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.901965 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.901932 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.902093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.902071 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.902167 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.902124 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.902270 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.902246 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.902486 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.902467 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6qs\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.902565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.902501 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:26.902625 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:26.902560 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.005221 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.005135 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.006149 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.006125 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.006387 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.006046 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.007315 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.007281 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.007665 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.006543 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.007866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.007846 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.008396 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.008377 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.008957 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.008799 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.009343 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.009320 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.009465 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.009376 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.009679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.009658 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6qs\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.010093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.010073 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.010194 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.010134 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.011702 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.010507 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.034814 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.020151 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.034814 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.023538 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.103245 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.103188 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6qs\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.105374 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.105346 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.389212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.389175 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:27.829456 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.829377 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh"] Nov 01 08:57:27.835048 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:57:27.835021 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode798ef27_c3a9_4502_b6d6_257386db3b28.slice/crio-8ac3fa535500c8d131cb61ffa23c1cb1265dd96cd68a080cad999e401c8ef198 WatchSource:0}: Error finding container 8ac3fa535500c8d131cb61ffa23c1cb1265dd96cd68a080cad999e401c8ef198: Status 404 returned error can't find the container with id 8ac3fa535500c8d131cb61ffa23c1cb1265dd96cd68a080cad999e401c8ef198 Nov 01 08:57:27.837702 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.837671 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:27.837811 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.837767 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:27.837811 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:27.837800 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:28.060004 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:28.059971 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" event={"ID":"e798ef27-c3a9-4502-b6d6-257386db3b28","Type":"ContainerStarted","Data":"53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63"} Nov 01 08:57:28.060197 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:28.060032 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" event={"ID":"e798ef27-c3a9-4502-b6d6-257386db3b28","Type":"ContainerStarted","Data":"8ac3fa535500c8d131cb61ffa23c1cb1265dd96cd68a080cad999e401c8ef198"} Nov 01 08:57:28.178640 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:28.178572 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podStartSLOduration=2.178548416 podStartE2EDuration="2.178548416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:28.17339617 +0000 UTC m=+1471.261732852" watchObservedRunningTime="2026-11-01 08:57:28.178548416 +0000 UTC m=+1471.266885086" Nov 01 08:57:28.392743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:28.390142 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:28.397765 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:28.397144 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:28.397765 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:28.397211 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:29.403742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:29.397292 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:29.403742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:29.397372 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:30.398745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:30.392174 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:30.398745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:30.392255 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:31.396213 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:31.396170 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:31.396902 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:31.396865 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:32.395962 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:32.395917 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:32.396268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:32.396234 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:33.395225 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:33.395182 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:33.395539 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:33.395509 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:34.389946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:34.389910 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 08:57:34.390347 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:34.389961 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 08:57:36.021836 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:36.021805 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:36.022422 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:36.022401 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:36.023338 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:36.023320 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 08:57:41.881587 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:41.881536 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w"] Nov 01 08:57:41.886647 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:41.886625 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.004663 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.004627 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.004859 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.004676 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.004919 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.004891 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.004961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.004932 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.005030 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.005015 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.005254 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.005240 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.005297 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.005282 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhpbq\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.005332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.005314 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.005451 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.005435 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.070698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.070652 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w"] Nov 01 08:57:42.106706 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.106665 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.106946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.106752 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhpbq\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.106946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.106778 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.106946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.106839 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.106946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.106900 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.106946 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.106940 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107241 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107007 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107241 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107033 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107241 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107078 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107241 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107187 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107581 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107600 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.107912 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.107689 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.108038 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.108014 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.109542 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.109522 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.109784 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.109765 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.165698 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.165626 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.165882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.165849 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhpbq\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq\") pod \"e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.202750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.202672 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:42.642143 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.642107 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w"] Nov 01 08:57:42.648140 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:57:42.648109 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86173b76_196a_4c9a_8e77_387714bb7ce6.slice/crio-4fef676e38c848d7c499eaf086f4021784d3ed88765a315268ff6f8dbdca7f17 WatchSource:0}: Error finding container 4fef676e38c848d7c499eaf086f4021784d3ed88765a315268ff6f8dbdca7f17: Status 404 returned error can't find the container with id 4fef676e38c848d7c499eaf086f4021784d3ed88765a315268ff6f8dbdca7f17 Nov 01 08:57:42.651639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.651604 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:42.651787 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.651677 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:42.651787 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:42.651706 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 08:57:43.287733 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:43.287691 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" event={"ID":"86173b76-196a-4c9a-8e77-387714bb7ce6","Type":"ContainerStarted","Data":"0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680"} Nov 01 08:57:43.288232 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:43.287780 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" event={"ID":"86173b76-196a-4c9a-8e77-387714bb7ce6","Type":"ContainerStarted","Data":"4fef676e38c848d7c499eaf086f4021784d3ed88765a315268ff6f8dbdca7f17"} Nov 01 08:57:43.450043 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:43.449969 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podStartSLOduration=2.44995253 podStartE2EDuration="2.44995253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:43.447420034 +0000 UTC m=+1486.535756739" watchObservedRunningTime="2026-11-01 08:57:43.44995253 +0000 UTC m=+1486.538289201" Nov 01 08:57:44.204464 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:44.204384 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:44.207512 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:44.207471 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:44.207696 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:44.207539 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:45.204266 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.204217 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:45.205615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.205537 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:45.610816 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.582994 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx"] Nov 01 08:57:45.814320 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.814282 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx"] Nov 01 08:57:45.845745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.821359 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:45.959915 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.933021 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d0ba8f\"" Nov 01 08:57:45.959915 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.940961 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:45.959915 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:45.941025 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzp5\" (UniqueName: \"kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5\") pod \"payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:46.123744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.082316 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:46.123744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.082437 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzp5\" (UniqueName: \"kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5\") pod \"payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:46.123744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.084597 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:46.123744 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:57:46.111980 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:57:46.182803 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.182757 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn"] Nov 01 08:57:46.245499 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.245001 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:46.245499 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.245018 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzp5\" (UniqueName: \"kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5\") pod \"payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:46.245499 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.245076 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:46.246660 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.246635 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn"] Nov 01 08:57:46.246865 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.246760 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.247844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.247823 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:46.309834 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.309792 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d0ba8f-dockercfg-7rq95\"" Nov 01 08:57:46.310260 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.310226 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr46b\" (UniqueName: \"kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b\") pod \"payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.310369 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.310302 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.411801 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.411759 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.412799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.412183 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr46b\" (UniqueName: \"kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b\") pod \"payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.412941 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.412829 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.475674 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.475637 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr46b\" (UniqueName: \"kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b\") pod \"payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.617796 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.617700 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:46.654466 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:46.653710 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx"] Nov 01 08:57:47.017328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.017239 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn"] Nov 01 08:57:47.023952 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:57:47.023922 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c57c63_9834_43b2_a18b_adff7d82abc0.slice/crio-4040893ad29411b16ca37b18ce15622b2bccd6689ff34704238f1729a36611f8 WatchSource:0}: Error finding container 4040893ad29411b16ca37b18ce15622b2bccd6689ff34704238f1729a36611f8: Status 404 returned error can't find the container with id 4040893ad29411b16ca37b18ce15622b2bccd6689ff34704238f1729a36611f8 Nov 01 08:57:47.204744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.204118 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:47.204744 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.204182 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:47.566925 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.566882 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" event={"ID":"b8c57c63-9834-43b2-a18b-adff7d82abc0","Type":"ContainerStarted","Data":"281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb"} Nov 01 08:57:47.567435 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.566943 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" event={"ID":"b8c57c63-9834-43b2-a18b-adff7d82abc0","Type":"ContainerStarted","Data":"4040893ad29411b16ca37b18ce15622b2bccd6689ff34704238f1729a36611f8"} Nov 01 08:57:47.567435 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.566967 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:47.569083 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.569049 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" event={"ID":"8a941e09-bdf9-451a-be9b-1c9db5bba1f0","Type":"ContainerStarted","Data":"e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1"} Nov 01 08:57:47.569242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.569111 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" event={"ID":"8a941e09-bdf9-451a-be9b-1c9db5bba1f0","Type":"ContainerStarted","Data":"56ea6433d22daae5ee18897e6d50c040dd4f1d0ad84809c84b3edcb2b4f9a87e"} Nov 01 08:57:47.569242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.569142 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:57:47.677571 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.677507 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" podStartSLOduration=2.677489855 podStartE2EDuration="2.677489855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:47.675409901 +0000 UTC m=+1490.763746573" watchObservedRunningTime="2026-11-01 08:57:47.677489855 +0000 UTC m=+1490.765826538" Nov 01 08:57:47.821795 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.786148 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" podStartSLOduration=2.786129807 podStartE2EDuration="2.786129807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:47.786047484 +0000 UTC m=+1490.874384157" watchObservedRunningTime="2026-11-01 08:57:47.786129807 +0000 UTC m=+1490.874466481" Nov 01 08:57:47.892746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.888472 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7409-3f1fc6"] Nov 01 08:57:47.976150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.948109 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:47.976150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:47.954645 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7409-3f1fc6"] Nov 01 08:57:48.028288 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.007992 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-44wvn\"" Nov 01 08:57:48.151773 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.149139 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz\") pod \"test-curl-7409-3f1fc6\" (UID: \"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550\") " pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:48.204183 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.204142 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:48.204488 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.204453 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:48.251682 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.250703 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz\") pod \"test-curl-7409-3f1fc6\" (UID: \"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550\") " pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:48.324826 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.324498 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz\") pod \"test-curl-7409-3f1fc6\" (UID: \"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550\") " pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:48.609319 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:48.609274 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:49.010109 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:49.010072 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7409-3f1fc6"] Nov 01 08:57:49.011690 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:57:49.011663 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod068fc8da_6f6e_4ea6_aa1e_3a37d71b9550.slice/crio-bdd53143d5762d49fafa5d1edb63ff65b60d84b43c2b3ba3b181c7fdb3ab48e5 WatchSource:0}: Error finding container bdd53143d5762d49fafa5d1edb63ff65b60d84b43c2b3ba3b181c7fdb3ab48e5: Status 404 returned error can't find the container with id bdd53143d5762d49fafa5d1edb63ff65b60d84b43c2b3ba3b181c7fdb3ab48e5 Nov 01 08:57:49.204034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:49.203990 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:49.204234 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:49.204071 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:49.585857 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:49.585812 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7409-3f1fc6" event={"ID":"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550","Type":"ContainerStarted","Data":"bdd53143d5762d49fafa5d1edb63ff65b60d84b43c2b3ba3b181c7fdb3ab48e5"} Nov 01 08:57:50.203236 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:50.203197 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 08:57:50.203652 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:50.203275 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 08:57:51.208110 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:51.208075 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:51.208506 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:51.208376 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:51.209268 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:51.209248 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 08:57:51.617692 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:51.617659 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7409-3f1fc6" event={"ID":"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550","Type":"ContainerStarted","Data":"3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268"} Nov 01 08:57:51.727010 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:51.726955 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7409-3f1fc6" podStartSLOduration=3.163493683 podStartE2EDuration="4.726937441s" totalImagesPullingTime="1.563443758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:57:51.723644258 +0000 UTC m=+1494.811980971" watchObservedRunningTime="2026-11-01 08:57:51.726937441 +0000 UTC m=+1494.815274112" Nov 01 08:57:53.255162 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:53.255115 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7409-3f1fc6"] Nov 01 08:57:53.260093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:53.260060 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7409-3f1fc6"] Nov 01 08:57:53.632117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:53.632076 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7409-3f1fc6" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" containerName="test-curl-7409-3f1fc6" containerID="cri-o://3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268" gracePeriod=2 Nov 01 08:57:53.684118 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:57:53.684073 5710 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7409-3f1fc6\" is forbidden: User \"system:node:ip-10-0-143-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-16.us-west-2.compute.internal' and this object" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:55.905346 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:55.905268 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:55.988917 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:55.988876 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz\") pod \"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550\" (UID: \"068fc8da-6f6e-4ea6-aa1e-3a37d71b9550\") " Nov 01 08:57:55.992405 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:55.992368 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz" pod "068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" (UID: "068fc8da-6f6e-4ea6-aa1e-3a37d71b9550"). InnerVolumeSpecName "kube-api-access-2xkdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:57:56.101472 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:56.101273 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xkdz\" (UniqueName: \"kubernetes.io/projected/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550-kube-api-access-2xkdz\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:57:56.786438 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:56.786398 5710 generic.go:356] "Generic (PLEG): container finished" podID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" containerID="3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268" exitCode=137 Nov 01 08:57:56.786853 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:56.786495 5710 scope.go:122] "RemoveContainer" containerID="3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268" Nov 01 08:57:56.787783 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:56.786532 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:56.801447 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:56.801422 5710 scope.go:122] "RemoveContainer" containerID="3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268" Nov 01 08:57:56.801923 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:57:56.801888 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268\": container with ID starting with 3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268 not found: ID does not exist" containerID="3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268" Nov 01 08:57:56.802036 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:56.801933 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268"} err="failed to get container status \"3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268\": rpc error: code = NotFound desc = could not find container \"3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268\": container with ID starting with 3f865eb9dc551719dc71cfa796bc51983280a93c013008da615f3c8cbaaec268 not found: ID does not exist" Nov 01 08:57:56.844910 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:57:56.844862 5710 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7409-3f1fc6\" is forbidden: User \"system:node:ip-10-0-143-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-16.us-west-2.compute.internal' and this object" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:57.060242 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:57.060153 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" path="/var/lib/kubelet/pods/068fc8da-6f6e-4ea6-aa1e-3a37d71b9550/volumes" Nov 01 08:57:57.102521 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:57:57.102479 5710 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7409-3f1fc6\" is forbidden: User \"system:node:ip-10-0-143-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-16.us-west-2.compute.internal' and this object" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" pod="opendatahub/test-curl-7409-3f1fc6" Nov 01 08:57:58.580117 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:58.580075 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 08:57:58.580582 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:57:58.580237 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 08:58:00.812316 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:00.812285 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:58:00.825487 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:00.825461 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 08:58:04.976745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:04.968812 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w"] Nov 01 08:58:04.999786 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:04.979143 5710 cpu_manager.go:414] "containerMap: removing container" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" containerName="test-curl-7409-3f1fc6" Nov 01 08:58:05.000155 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.000126 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" containerName="test-curl-7409-3f1fc6" Nov 01 08:58:05.000679 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.000656 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="068fc8da-6f6e-4ea6-aa1e-3a37d71b9550" containerName="test-curl-7409-3f1fc6" Nov 01 08:58:05.031747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.027509 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.190846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.175693 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w"] Nov 01 08:58:05.190846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.185831 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-03832c\"" Nov 01 08:58:05.190846 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.186090 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-03832c-dockercfg-vdbz6\"" Nov 01 08:58:05.278803 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.261431 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smz5p\" (UniqueName: \"kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p\") pod \"payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.288771 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.279257 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.444122 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.444071 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smz5p\" (UniqueName: \"kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p\") pod \"payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.444945 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.444890 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.446307 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.446281 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.502743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.502694 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smz5p\" (UniqueName: \"kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p\") pod \"payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:05.799339 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:05.799294 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:06.277358 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:06.267782 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:58:06.359531 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:06.359502 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w"] Nov 01 08:58:07.326778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:07.325704 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" event={"ID":"3e11f3da-7f16-429c-a72b-0978781d6cdc","Type":"ContainerStarted","Data":"4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9"} Nov 01 08:58:07.326778 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:07.325819 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" event={"ID":"3e11f3da-7f16-429c-a72b-0978781d6cdc","Type":"ContainerStarted","Data":"ae8e2ba96553b4bc4a8afea9f4b07e2b6e6703838ad292702dcad939b71ac9e9"} Nov 01 08:58:07.327401 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:07.327382 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:07.452745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:07.447803 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" podStartSLOduration=3.447785911 podStartE2EDuration="3.447785911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:07.446957887 +0000 UTC m=+1510.535294559" watchObservedRunningTime="2026-11-01 08:58:07.447785911 +0000 UTC m=+1510.536122586" Nov 01 08:58:15.480752 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:58:15.480695 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:58:19.438811 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:19.438758 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 08:58:31.993074 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:31.993034 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj"] Nov 01 08:58:32.092369 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:32.092329 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl"] Nov 01 08:58:35.356913 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.356828 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7"] Nov 01 08:58:35.362978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.362952 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.381394 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.381357 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8rd\" (UniqueName: \"kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.381606 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.381457 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.381805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.381776 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.381906 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.381886 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.416114 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.416074 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7"] Nov 01 08:58:35.457808 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.457766 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d0ba8f-serving-cert\"" Nov 01 08:58:35.484827 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.484790 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.485166 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.485140 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.485414 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.485380 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8rd\" (UniqueName: \"kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.485564 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.485510 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.485646 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.485580 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.485710 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:58:35.485691 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-serving-cert: secret "maas-api-e2e-api-b-d0ba8f-serving-cert" not found Nov 01 08:58:35.485815 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:58:35.485802 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls podName:b087af6c-ac01-4251-8210-913b3e18122e nodeName:}" failed. No retries permitted until 2026-11-01 08:58:35.985780292 +0000 UTC m=+1539.074116959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls") pod "maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" (UID: "b087af6c-ac01-4251-8210-913b3e18122e") : secret "maas-api-e2e-api-b-d0ba8f-serving-cert" not found Nov 01 08:58:35.490573 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.490547 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.545892 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.545849 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8rd\" (UniqueName: \"kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:35.992011 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:35.991964 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:36.029444 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:36.018413 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\") pod \"maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:36.283710 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:36.283607 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:36.764978 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:36.718926 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7"] Nov 01 08:58:37.003057 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.002948 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" podUID="ce0016e9-e03b-433a-a75e-c3f546a53c77" containerName="payload-processing" containerID="cri-o://77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213" gracePeriod=25 Nov 01 08:58:37.083424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.083022 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:58:37.083424 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.083106 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:58:37.106812 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.098808 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" podUID="d18cf697-e205-410b-bb97-bb56b8b6ff81" containerName="payload-pre-processing" containerID="cri-o://93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e" gracePeriod=25 Nov 01 08:58:37.180639 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.180430 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" event={"ID":"b087af6c-ac01-4251-8210-913b3e18122e","Type":"ContainerStarted","Data":"1cb1f98a4d0e3954984acb79ce3985e4b7dd956a143891110cb9479a93efd549"} Nov 01 08:58:37.254801 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.252867 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 08:58:37.254801 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.252939 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 08:58:37.923868 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:37.923835 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:58:38.020954 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.020928 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:58:38.051627 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.051598 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn\" (UniqueName: \"kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn\") pod \"ce0016e9-e03b-433a-a75e-c3f546a53c77\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " Nov 01 08:58:38.051627 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.051645 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume\") pod \"d18cf697-e205-410b-bb97-bb56b8b6ff81\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " Nov 01 08:58:38.051961 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.051833 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28\" (UniqueName: \"kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28\") pod \"d18cf697-e205-410b-bb97-bb56b8b6ff81\" (UID: \"d18cf697-e205-410b-bb97-bb56b8b6ff81\") " Nov 01 08:58:38.052036 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.052000 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume\") pod \"ce0016e9-e03b-433a-a75e-c3f546a53c77\" (UID: \"ce0016e9-e03b-433a-a75e-c3f546a53c77\") " Nov 01 08:58:38.052227 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.052174 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume" pod "d18cf697-e205-410b-bb97-bb56b8b6ff81" (UID: "d18cf697-e205-410b-bb97-bb56b8b6ff81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:58:38.052402 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.052377 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume" pod "ce0016e9-e03b-433a-a75e-c3f546a53c77" (UID: "ce0016e9-e03b-433a-a75e-c3f546a53c77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:58:38.052876 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.052856 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0016e9-e03b-433a-a75e-c3f546a53c77-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:58:38.052972 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.052880 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18cf697-e205-410b-bb97-bb56b8b6ff81-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:58:38.054289 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.054263 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn" pod "ce0016e9-e03b-433a-a75e-c3f546a53c77" (UID: "ce0016e9-e03b-433a-a75e-c3f546a53c77"). InnerVolumeSpecName "kube-api-access-zsjdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:58:38.054467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.054447 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28" pod "d18cf697-e205-410b-bb97-bb56b8b6ff81" (UID: "d18cf697-e205-410b-bb97-bb56b8b6ff81"). InnerVolumeSpecName "kube-api-access-n8r28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:58:38.155023 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.154930 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8r28\" (UniqueName: \"kubernetes.io/projected/d18cf697-e205-410b-bb97-bb56b8b6ff81-kube-api-access-n8r28\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:58:38.155023 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.154967 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsjdn\" (UniqueName: \"kubernetes.io/projected/ce0016e9-e03b-433a-a75e-c3f546a53c77-kube-api-access-zsjdn\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:58:38.176902 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.176857 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" event={"ID":"b087af6c-ac01-4251-8210-913b3e18122e","Type":"ContainerStarted","Data":"93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4"} Nov 01 08:58:38.177094 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.176932 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:38.178544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.178519 5710 generic.go:356] "Generic (PLEG): container finished" podID="ce0016e9-e03b-433a-a75e-c3f546a53c77" containerID="77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213" exitCode=0 Nov 01 08:58:38.178675 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.178574 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" Nov 01 08:58:38.178675 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.178602 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" event={"ID":"ce0016e9-e03b-433a-a75e-c3f546a53c77","Type":"ContainerDied","Data":"77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213"} Nov 01 08:58:38.178675 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.178651 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj" event={"ID":"ce0016e9-e03b-433a-a75e-c3f546a53c77","Type":"ContainerDied","Data":"21d3de00f792c040ecbe838f167980c03ead724f56b0cb6d28633bf2bca095fc"} Nov 01 08:58:38.178902 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.178680 5710 scope.go:122] "RemoveContainer" containerID="77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213" Nov 01 08:58:38.180153 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.180132 5710 generic.go:356] "Generic (PLEG): container finished" podID="d18cf697-e205-410b-bb97-bb56b8b6ff81" containerID="93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e" exitCode=0 Nov 01 08:58:38.180269 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.180177 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" Nov 01 08:58:38.180269 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.180194 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" event={"ID":"d18cf697-e205-410b-bb97-bb56b8b6ff81","Type":"ContainerDied","Data":"93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e"} Nov 01 08:58:38.180360 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.180289 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl" event={"ID":"d18cf697-e205-410b-bb97-bb56b8b6ff81","Type":"ContainerDied","Data":"e11479bfc506a62ed315e2a8fed3ec556480a7c0c671a263ebd3b503842c5300"} Nov 01 08:58:38.191548 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.191508 5710 scope.go:122] "RemoveContainer" containerID="77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213" Nov 01 08:58:38.193474 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:58:38.193437 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213\": container with ID starting with 77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213 not found: ID does not exist" containerID="77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213" Nov 01 08:58:38.193612 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.193484 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213"} err="failed to get container status \"77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213\": rpc error: code = NotFound desc = could not find container \"77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213\": container with ID starting with 77747925d2f091fe8c5a58f67ac4111e8dec479314f5e55550d9328b66503213 not found: ID does not exist" Nov 01 08:58:38.193612 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.193512 5710 scope.go:122] "RemoveContainer" containerID="93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e" Nov 01 08:58:38.205281 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.205263 5710 scope.go:122] "RemoveContainer" containerID="93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e" Nov 01 08:58:38.205508 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:58:38.205485 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e\": container with ID starting with 93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e not found: ID does not exist" containerID="93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e" Nov 01 08:58:38.205563 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.205520 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e"} err="failed to get container status \"93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e\": rpc error: code = NotFound desc = could not find container \"93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e\": container with ID starting with 93789b5ed597138bc7eb8767781161d780c207dffcb14e75f360fa6c5f44296e not found: ID does not exist" Nov 01 08:58:38.290399 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.290334 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" podStartSLOduration=2.16169355 podStartE2EDuration="3.290318747s" totalImagesPullingTime="1.128625197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:38.289283278 +0000 UTC m=+1541.377620040" watchObservedRunningTime="2026-11-01 08:58:38.290318747 +0000 UTC m=+1541.378655419" Nov 01 08:58:38.397673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.397624 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj"] Nov 01 08:58:38.455079 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.454970 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93665b-5d94689985-srgzj"] Nov 01 08:58:38.546053 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.546015 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl"] Nov 01 08:58:38.551333 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:38.551300 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93665b-858fcb9987-fw9sl"] Nov 01 08:58:39.060339 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:39.060309 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0016e9-e03b-433a-a75e-c3f546a53c77" path="/var/lib/kubelet/pods/ce0016e9-e03b-433a-a75e-c3f546a53c77/volumes" Nov 01 08:58:39.060862 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:39.060844 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18cf697-e205-410b-bb97-bb56b8b6ff81" path="/var/lib/kubelet/pods/d18cf697-e205-410b-bb97-bb56b8b6ff81/volumes" Nov 01 08:58:44.194987 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:44.194960 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 08:58:45.479127 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:58:45.479087 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:58:46.433896 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.433856 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm"] Nov 01 08:58:46.434998 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.434972 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d18cf697-e205-410b-bb97-bb56b8b6ff81" containerName="payload-pre-processing" Nov 01 08:58:46.434998 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.434999 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18cf697-e205-410b-bb97-bb56b8b6ff81" containerName="payload-pre-processing" Nov 01 08:58:46.435150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.435068 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ce0016e9-e03b-433a-a75e-c3f546a53c77" containerName="payload-processing" Nov 01 08:58:46.435150 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.435076 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0016e9-e03b-433a-a75e-c3f546a53c77" containerName="payload-processing" Nov 01 08:58:46.435270 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.435257 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0016e9-e03b-433a-a75e-c3f546a53c77" containerName="payload-processing" Nov 01 08:58:46.435301 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.435280 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18cf697-e205-410b-bb97-bb56b8b6ff81" containerName="payload-pre-processing" Nov 01 08:58:46.439931 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.439907 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:46.491246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.491205 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm"] Nov 01 08:58:46.577742 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.543490 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3901b8\"" Nov 01 08:58:46.604743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.590042 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3901b8-dockercfg-ngh6f\"" Nov 01 08:58:46.715799 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.656689 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx"] Nov 01 08:58:46.784743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.783829 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:46.796739 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.787946 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xpf\" (UniqueName: \"kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf\") pod \"payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:46.875193 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.848141 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:46.907429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.907289 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3901b8-558886c749-p9fjx\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:46.907429 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.907344 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bml2j\" (UniqueName: \"kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j\") pod \"payload-processing-e2e-api-a-3901b8-558886c749-p9fjx\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:46.908073 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.908010 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xpf\" (UniqueName: \"kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf\") pod \"payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:46.908467 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.908394 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:46.910034 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.910010 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:46.920874 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.920839 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx"] Nov 01 08:58:46.971335 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:46.971261 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xpf\" (UniqueName: \"kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf\") pod \"payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:47.012331 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.012295 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3901b8-558886c749-p9fjx\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:47.012522 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.012341 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bml2j\" (UniqueName: \"kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j\") pod \"payload-processing-e2e-api-a-3901b8-558886c749-p9fjx\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:47.013212 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.013185 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3901b8-558886c749-p9fjx\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:47.056417 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.056379 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:47.070535 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.070503 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bml2j\" (UniqueName: \"kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j\") pod \"payload-processing-e2e-api-a-3901b8-558886c749-p9fjx\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:47.239820 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.239700 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:47.453872 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.453828 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm"] Nov 01 08:58:47.461811 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:58:47.461779 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcd841e_5e8c_4df1_9d22_7ab1624d3c8c.slice/crio-f34c0a635b9ec61b03d414336bb5190f4b8cbea9baa8484aef17fb8bed0a99b3 WatchSource:0}: Error finding container f34c0a635b9ec61b03d414336bb5190f4b8cbea9baa8484aef17fb8bed0a99b3: Status 404 returned error can't find the container with id f34c0a635b9ec61b03d414336bb5190f4b8cbea9baa8484aef17fb8bed0a99b3 Nov 01 08:58:47.632169 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:47.632097 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx"] Nov 01 08:58:47.636762 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:58:47.636734 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac2844b_8082_4577_b118_e785d36a469b.slice/crio-128182e6245b47eed8768e53f1b39844a4155796b93f3768624fc105637e8453 WatchSource:0}: Error finding container 128182e6245b47eed8768e53f1b39844a4155796b93f3768624fc105637e8453: Status 404 returned error can't find the container with id 128182e6245b47eed8768e53f1b39844a4155796b93f3768624fc105637e8453 Nov 01 08:58:48.276693 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.276647 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" event={"ID":"fac2844b-8082-4577-b118-e785d36a469b","Type":"ContainerStarted","Data":"075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832"} Nov 01 08:58:48.276942 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.276760 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:48.276942 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.276787 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" event={"ID":"fac2844b-8082-4577-b118-e785d36a469b","Type":"ContainerStarted","Data":"128182e6245b47eed8768e53f1b39844a4155796b93f3768624fc105637e8453"} Nov 01 08:58:48.278462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.278430 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" event={"ID":"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c","Type":"ContainerStarted","Data":"dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9"} Nov 01 08:58:48.278621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.278479 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" event={"ID":"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c","Type":"ContainerStarted","Data":"f34c0a635b9ec61b03d414336bb5190f4b8cbea9baa8484aef17fb8bed0a99b3"} Nov 01 08:58:48.278621 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.278527 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:58:48.388889 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:48.388835 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" podStartSLOduration=2.388818882 podStartE2EDuration="2.388818882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:48.385031093 +0000 UTC m=+1551.473367767" watchObservedRunningTime="2026-11-01 08:58:48.388818882 +0000 UTC m=+1551.477155554" Nov 01 08:58:59.196866 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:59.192165 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" podStartSLOduration=13.192146313 podStartE2EDuration="13.192146313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:58:48.504076636 +0000 UTC m=+1551.592413311" watchObservedRunningTime="2026-11-01 08:58:59.192146313 +0000 UTC m=+1562.280482986" Nov 01 08:58:59.263746 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:59.263463 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:58:59.290329 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:59.290295 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 08:58:59.291100 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:58:59.291076 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 08:59:04.048327 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:04.048285 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j"] Nov 01 08:59:04.049116 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:04.049084 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" containerID="cri-o://d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e" gracePeriod=30 Nov 01 08:59:06.951233 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:06.951193 5710 patch_prober.go:29] interesting pod/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 08:59:06.951809 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:06.951265 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 08:59:09.337844 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.337817 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:59:09.420882 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.420838 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.420941 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421002 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421053 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421201 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421124 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421201 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421156 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421191 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:59:09.421300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421210 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421230 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:59:09.421300 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421258 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421392 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:59:09.421509 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421398 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert\") pod \"0ce47b1f-78db-4688-a11a-2ed9daa86803\" (UID: \"0ce47b1f-78db-4688-a11a-2ed9daa86803\") " Nov 01 08:59:09.421611 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421591 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:59:09.421781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.421758 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:59:09.422198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.422178 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.422198 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.422199 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.422336 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.422208 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.422336 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.422218 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce47b1f-78db-4688-a11a-2ed9daa86803-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.422336 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.422231 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.423750 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.423711 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 08:59:09.423838 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.423743 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "kube-api-access-w84tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:09.424039 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.424007 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 08:59:09.424039 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.424025 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token" pod "0ce47b1f-78db-4688-a11a-2ed9daa86803" (UID: "0ce47b1f-78db-4688-a11a-2ed9daa86803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:09.523816 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.523783 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84tm\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-kube-api-access-w84tm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.523816 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.523815 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.524095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.523832 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.524095 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.523847 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce47b1f-78db-4688-a11a-2ed9daa86803-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:09.591989 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.591956 5710 generic.go:356] "Generic (PLEG): container finished" podID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerID="d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e" exitCode=0 Nov 01 08:59:09.592168 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.592009 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" Nov 01 08:59:09.592168 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.592035 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" event={"ID":"0ce47b1f-78db-4688-a11a-2ed9daa86803","Type":"ContainerDied","Data":"d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e"} Nov 01 08:59:09.592168 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.592090 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j" event={"ID":"0ce47b1f-78db-4688-a11a-2ed9daa86803","Type":"ContainerDied","Data":"5e0665813637dd1cc4a5cc3afd14c71ff1aadc376d976b5a7e698f6b78decc67"} Nov 01 08:59:09.592168 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.592115 5710 scope.go:122] "RemoveContainer" containerID="d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e" Nov 01 08:59:09.603980 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.603960 5710 scope.go:122] "RemoveContainer" containerID="61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063" Nov 01 08:59:09.617328 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.617302 5710 scope.go:122] "RemoveContainer" containerID="d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e" Nov 01 08:59:09.617685 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:09.617659 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e\": container with ID starting with d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e not found: ID does not exist" containerID="d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e" Nov 01 08:59:09.617828 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.617689 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e"} err="failed to get container status \"d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e\": rpc error: code = NotFound desc = could not find container \"d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e\": container with ID starting with d497786f40a1436ebcedf27f03640266c9d586340eea418f0b384f7d7aa3870e not found: ID does not exist" Nov 01 08:59:09.617828 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.617709 5710 scope.go:122] "RemoveContainer" containerID="61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063" Nov 01 08:59:09.618106 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:09.618086 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063\": container with ID starting with 61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063 not found: ID does not exist" containerID="61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063" Nov 01 08:59:09.618146 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.618115 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063"} err="failed to get container status \"61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063\": rpc error: code = NotFound desc = could not find container \"61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063\": container with ID starting with 61c55c2805136b5fc64125cfbd4039dcb9f7d0228897f84af43cf892babb8063 not found: ID does not exist" Nov 01 08:59:09.749070 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.749041 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j"] Nov 01 08:59:09.752821 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:09.752778 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-93665b-openshift-default-79fd8998dc-54s7j"] Nov 01 08:59:11.059890 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:11.059854 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" path="/var/lib/kubelet/pods/0ce47b1f-78db-4688-a11a-2ed9daa86803/volumes" Nov 01 08:59:11.448056 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:11.448016 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89"] Nov 01 08:59:11.547397 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:11.547365 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k"] Nov 01 08:59:15.479865 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:15.479815 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:59:16.448377 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.448320 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" podUID="0b2ceec5-21ce-4cd2-aec1-edea0872f31d" containerName="payload-processing" containerID="cri-o://582c8fe4361c498d66f0252f0967be8e3c40ddaaa199e8789f93557b6f8b303b" gracePeriod=25 Nov 01 08:59:16.548575 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.548528 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" podUID="e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" containerName="payload-pre-processing" containerID="cri-o://802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee" gracePeriod=25 Nov 01 08:59:16.662781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.657480 5710 generic.go:356] "Generic (PLEG): container finished" podID="0b2ceec5-21ce-4cd2-aec1-edea0872f31d" containerID="582c8fe4361c498d66f0252f0967be8e3c40ddaaa199e8789f93557b6f8b303b" exitCode=0 Nov 01 08:59:16.662781 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.657555 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" event={"ID":"0b2ceec5-21ce-4cd2-aec1-edea0872f31d","Type":"ContainerDied","Data":"582c8fe4361c498d66f0252f0967be8e3c40ddaaa199e8789f93557b6f8b303b"} Nov 01 08:59:16.760624 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.760588 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:59:16.872856 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.871780 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh\" (UniqueName: \"kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh\") pod \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " Nov 01 08:59:16.872856 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.871994 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume\") pod \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\" (UID: \"0b2ceec5-21ce-4cd2-aec1-edea0872f31d\") " Nov 01 08:59:16.872856 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.872428 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume" pod "0b2ceec5-21ce-4cd2-aec1-edea0872f31d" (UID: "0b2ceec5-21ce-4cd2-aec1-edea0872f31d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:59:16.872856 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.872822 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:16.874906 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.874879 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh" pod "0b2ceec5-21ce-4cd2-aec1-edea0872f31d" (UID: "0b2ceec5-21ce-4cd2-aec1-edea0872f31d"). InnerVolumeSpecName "kube-api-access-5fzgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:16.883351 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.883321 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:59:16.973805 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.973770 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume\") pod \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " Nov 01 08:59:16.973984 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.973914 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl\" (UniqueName: \"kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl\") pod \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\" (UID: \"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0\") " Nov 01 08:59:16.974202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.974153 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume" pod "e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" (UID: "e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:59:16.974673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.974648 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:16.974673 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.974674 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fzgh\" (UniqueName: \"kubernetes.io/projected/0b2ceec5-21ce-4cd2-aec1-edea0872f31d-kube-api-access-5fzgh\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:16.976228 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:16.976201 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl" pod "e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" (UID: "e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0"). InnerVolumeSpecName "kube-api-access-qv8fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:17.075864 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.075835 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv8fl\" (UniqueName: \"kubernetes.io/projected/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0-kube-api-access-qv8fl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:17.667525 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.667493 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" Nov 01 08:59:17.667525 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.667526 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89" event={"ID":"0b2ceec5-21ce-4cd2-aec1-edea0872f31d","Type":"ContainerDied","Data":"9834554d1d4ca0e5a5e6c8c5327c5786c00205345048ab7f959ce941d39979f3"} Nov 01 08:59:17.668108 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.667595 5710 scope.go:122] "RemoveContainer" containerID="582c8fe4361c498d66f0252f0967be8e3c40ddaaa199e8789f93557b6f8b303b" Nov 01 08:59:17.669326 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.669298 5710 generic.go:356] "Generic (PLEG): container finished" podID="e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" containerID="802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee" exitCode=0 Nov 01 08:59:17.669450 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.669361 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" Nov 01 08:59:17.669450 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.669395 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" event={"ID":"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0","Type":"ContainerDied","Data":"802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee"} Nov 01 08:59:17.669553 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.669457 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k" event={"ID":"e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0","Type":"ContainerDied","Data":"74698ff400c1b7e8d87b6366bbaa2568b1aa3c76c133a8ee5cdfe38ec1cc22d4"} Nov 01 08:59:17.679558 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.679535 5710 scope.go:122] "RemoveContainer" containerID="802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee" Nov 01 08:59:17.692173 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.692122 5710 scope.go:122] "RemoveContainer" containerID="802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee" Nov 01 08:59:17.692572 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:17.692553 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee\": container with ID starting with 802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee not found: ID does not exist" containerID="802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee" Nov 01 08:59:17.692640 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.692582 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee"} err="failed to get container status \"802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee\": rpc error: code = NotFound desc = could not find container \"802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee\": container with ID starting with 802fa855c8469594142b7436e37a93467a07d4ac6bbd4f8f76d8baff0c378eee not found: ID does not exist" Nov 01 08:59:17.825272 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.825161 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k"] Nov 01 08:59:17.871145 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.871110 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8701a0-66c6d75d7-sk59k"] Nov 01 08:59:17.932020 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.931982 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89"] Nov 01 08:59:17.936676 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:17.936644 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8701a0-6d98f58c7f-88x89"] Nov 01 08:59:19.060184 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:19.060145 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2ceec5-21ce-4cd2-aec1-edea0872f31d" path="/var/lib/kubelet/pods/0b2ceec5-21ce-4cd2-aec1-edea0872f31d/volumes" Nov 01 08:59:19.060885 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:19.060862 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" path="/var/lib/kubelet/pods/e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0/volumes" Nov 01 08:59:32.434469 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.434426 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx"] Nov 01 08:59:32.435246 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435229 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0b2ceec5-21ce-4cd2-aec1-edea0872f31d" containerName="payload-processing" Nov 01 08:59:32.435312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435247 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2ceec5-21ce-4cd2-aec1-edea0872f31d" containerName="payload-processing" Nov 01 08:59:32.435312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435262 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" Nov 01 08:59:32.435312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435268 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" Nov 01 08:59:32.435312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435289 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" Nov 01 08:59:32.435312 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435294 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" Nov 01 08:59:32.435458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435316 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" containerName="payload-pre-processing" Nov 01 08:59:32.435458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435321 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" containerName="payload-pre-processing" Nov 01 08:59:32.435458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435430 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2ceec5-21ce-4cd2-aec1-edea0872f31d" containerName="payload-processing" Nov 01 08:59:32.435458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435439 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" Nov 01 08:59:32.435458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435449 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce47b1f-78db-4688-a11a-2ed9daa86803" containerName="istio-proxy" Nov 01 08:59:32.435458 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.435457 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e435d4f3-3c46-403b-aff4-cd3cf0b2c3c0" containerName="payload-pre-processing" Nov 01 08:59:32.441001 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.440976 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.489939 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.489905 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3901b8-serving-cert\"" Nov 01 08:59:32.493544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.493515 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx"] Nov 01 08:59:32.570258 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.570220 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9wk\" (UniqueName: \"kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.570519 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.570499 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.570560 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.570549 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.570619 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.570606 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.672219 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.672176 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.672437 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.672349 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9wk\" (UniqueName: \"kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.672578 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.672561 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.672649 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.672621 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.672974 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.672952 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.675544 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.675515 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.675661 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.675515 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.730350 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.730271 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9wk\" (UniqueName: \"kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk\") pod \"maas-api-e2e-api-a-3901b8-56698f4944-l4rgx\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:32.756541 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:32.756486 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:33.342510 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:33.342469 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx"] Nov 01 08:59:33.349282 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:59:33.349242 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646ffb4c_22dd_4e93_b4c9_5ed44f35fe1b.slice/crio-9157195d6a0ef58c0052ad965cbb76d34bec9fca75b693c0c35e3ded40b96785 WatchSource:0}: Error finding container 9157195d6a0ef58c0052ad965cbb76d34bec9fca75b693c0c35e3ded40b96785: Status 404 returned error can't find the container with id 9157195d6a0ef58c0052ad965cbb76d34bec9fca75b693c0c35e3ded40b96785 Nov 01 08:59:33.834415 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:33.834326 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" event={"ID":"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b","Type":"ContainerStarted","Data":"9157195d6a0ef58c0052ad965cbb76d34bec9fca75b693c0c35e3ded40b96785"} Nov 01 08:59:34.844250 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:34.844206 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" event={"ID":"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b","Type":"ContainerStarted","Data":"f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e"} Nov 01 08:59:34.844788 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:34.844272 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:34.955112 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:34.955052 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" podStartSLOduration=2.008686113 podStartE2EDuration="2.955034406s" totalImagesPullingTime="946.348293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:59:34.954610981 +0000 UTC m=+1598.042947652" watchObservedRunningTime="2026-11-01 08:59:34.955034406 +0000 UTC m=+1598.043371078" Nov 01 08:59:40.858370 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:40.858335 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 08:59:41.980594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:41.980561 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5"] Nov 01 08:59:41.981103 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:41.980795 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" podUID="3ffff0af-66fc-4b11-9853-60ab0709a496" containerName="maas-api" containerID="cri-o://e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4" gracePeriod=30 Nov 01 08:59:42.372528 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.372475 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62"] Nov 01 08:59:42.400952 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.400926 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:59:42.470514 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.470470 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs\") pod \"3ffff0af-66fc-4b11-9853-60ab0709a496\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " Nov 01 08:59:42.470752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.470675 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls\") pod \"3ffff0af-66fc-4b11-9853-60ab0709a496\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " Nov 01 08:59:42.470806 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.470765 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca\") pod \"3ffff0af-66fc-4b11-9853-60ab0709a496\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " Nov 01 08:59:42.471017 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.470990 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp\" (UniqueName: \"kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp\") pod \"3ffff0af-66fc-4b11-9853-60ab0709a496\" (UID: \"3ffff0af-66fc-4b11-9853-60ab0709a496\") " Nov 01 08:59:42.471217 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.471182 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca" pod "3ffff0af-66fc-4b11-9853-60ab0709a496" (UID: "3ffff0af-66fc-4b11-9853-60ab0709a496"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:59:42.471660 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.471641 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffff0af-66fc-4b11-9853-60ab0709a496-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:42.473359 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.473328 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp" pod "3ffff0af-66fc-4b11-9853-60ab0709a496" (UID: "3ffff0af-66fc-4b11-9853-60ab0709a496"). InnerVolumeSpecName "kube-api-access-8lwwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:42.473568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.473541 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs" pod "3ffff0af-66fc-4b11-9853-60ab0709a496" (UID: "3ffff0af-66fc-4b11-9853-60ab0709a496"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:59:42.473568 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.473557 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls" pod "3ffff0af-66fc-4b11-9853-60ab0709a496" (UID: "3ffff0af-66fc-4b11-9853-60ab0709a496"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 08:59:42.474323 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.474302 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7"] Nov 01 08:59:42.572741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.572625 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:42.572741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.572662 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lwwp\" (UniqueName: \"kubernetes.io/projected/3ffff0af-66fc-4b11-9853-60ab0709a496-kube-api-access-8lwwp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:42.572741 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.572677 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ffff0af-66fc-4b11-9853-60ab0709a496-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:42.986829 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.986789 5710 generic.go:356] "Generic (PLEG): container finished" podID="3ffff0af-66fc-4b11-9853-60ab0709a496" containerID="e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4" exitCode=0 Nov 01 08:59:42.987280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.986833 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" Nov 01 08:59:42.987280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.986875 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" event={"ID":"3ffff0af-66fc-4b11-9853-60ab0709a496","Type":"ContainerDied","Data":"e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4"} Nov 01 08:59:42.987280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.986932 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5" event={"ID":"3ffff0af-66fc-4b11-9853-60ab0709a496","Type":"ContainerDied","Data":"6b66e3178b8c4a0d84efc3aeba5bdff1c3b6964bbe57fec2fa968fe19e25e138"} Nov 01 08:59:42.987280 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:42.986959 5710 scope.go:122] "RemoveContainer" containerID="e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4" Nov 01 08:59:43.009013 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:43.008986 5710 scope.go:122] "RemoveContainer" containerID="e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4" Nov 01 08:59:43.009443 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:43.009419 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4\": container with ID starting with e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4 not found: ID does not exist" containerID="e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4" Nov 01 08:59:43.009541 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:43.009456 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4"} err="failed to get container status \"e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4\": rpc error: code = NotFound desc = could not find container \"e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4\": container with ID starting with e66620ff99e2a39ca98a4f179896c956b4a0c4cc648b7622cd8ca0882cc60fa4 not found: ID does not exist" Nov 01 08:59:43.297088 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:43.296950 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5"] Nov 01 08:59:43.297366 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:43.297297 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-z8gb5"] Nov 01 08:59:44.557500 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:44.557234 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 08:59:45.061350 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:45.061312 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffff0af-66fc-4b11-9853-60ab0709a496" path="/var/lib/kubelet/pods/3ffff0af-66fc-4b11-9853-60ab0709a496/volumes" Nov 01 08:59:45.483201 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:45.483154 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 08:59:47.373365 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.373314 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" podUID="1645573a-e4c5-42f5-8671-9c68931b532c" containerName="payload-processing" containerID="cri-o://6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5" gracePeriod=25 Nov 01 08:59:47.474851 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.474792 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" podUID="53a7f922-e7f1-4aa3-9858-7acddab9c950" containerName="payload-pre-processing" containerID="cri-o://ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8" gracePeriod=25 Nov 01 08:59:47.694361 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.694328 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:59:47.718854 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.718817 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume\") pod \"1645573a-e4c5-42f5-8671-9c68931b532c\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " Nov 01 08:59:47.719026 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.718968 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w\" (UniqueName: \"kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w\") pod \"1645573a-e4c5-42f5-8671-9c68931b532c\" (UID: \"1645573a-e4c5-42f5-8671-9c68931b532c\") " Nov 01 08:59:47.720656 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.720438 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume" pod "1645573a-e4c5-42f5-8671-9c68931b532c" (UID: "1645573a-e4c5-42f5-8671-9c68931b532c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:59:47.723058 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.723025 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w" pod "1645573a-e4c5-42f5-8671-9c68931b532c" (UID: "1645573a-e4c5-42f5-8671-9c68931b532c"). InnerVolumeSpecName "kube-api-access-rbj6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:47.761565 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.761538 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:59:47.821118 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.821078 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume\") pod \"53a7f922-e7f1-4aa3-9858-7acddab9c950\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " Nov 01 08:59:47.821309 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.821223 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd\" (UniqueName: \"kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd\") pod \"53a7f922-e7f1-4aa3-9858-7acddab9c950\" (UID: \"53a7f922-e7f1-4aa3-9858-7acddab9c950\") " Nov 01 08:59:47.821574 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.821542 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume" pod "53a7f922-e7f1-4aa3-9858-7acddab9c950" (UID: "53a7f922-e7f1-4aa3-9858-7acddab9c950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 08:59:47.822021 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.821993 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1645573a-e4c5-42f5-8671-9c68931b532c-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:47.822152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.822024 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbj6w\" (UniqueName: \"kubernetes.io/projected/1645573a-e4c5-42f5-8671-9c68931b532c-kube-api-access-rbj6w\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:47.822152 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.822040 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a7f922-e7f1-4aa3-9858-7acddab9c950-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:47.823641 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.823612 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd" pod "53a7f922-e7f1-4aa3-9858-7acddab9c950" (UID: "53a7f922-e7f1-4aa3-9858-7acddab9c950"). InnerVolumeSpecName "kube-api-access-5p4zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 08:59:47.923712 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:47.923614 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4zd\" (UniqueName: \"kubernetes.io/projected/53a7f922-e7f1-4aa3-9858-7acddab9c950-kube-api-access-5p4zd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 08:59:48.104404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.104368 5710 generic.go:356] "Generic (PLEG): container finished" podID="1645573a-e4c5-42f5-8671-9c68931b532c" containerID="6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5" exitCode=0 Nov 01 08:59:48.104615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.104430 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" Nov 01 08:59:48.104615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.104460 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" event={"ID":"1645573a-e4c5-42f5-8671-9c68931b532c","Type":"ContainerDied","Data":"6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5"} Nov 01 08:59:48.104615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.104525 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62" event={"ID":"1645573a-e4c5-42f5-8671-9c68931b532c","Type":"ContainerDied","Data":"6e8976eb38479f7329c6aa4717dc0177f8dee66c60672ef944e60a6560035cc8"} Nov 01 08:59:48.104615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.104559 5710 scope.go:122] "RemoveContainer" containerID="6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5" Nov 01 08:59:48.106210 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.106180 5710 generic.go:356] "Generic (PLEG): container finished" podID="53a7f922-e7f1-4aa3-9858-7acddab9c950" containerID="ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8" exitCode=0 Nov 01 08:59:48.106471 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.106222 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" Nov 01 08:59:48.106471 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.106244 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" event={"ID":"53a7f922-e7f1-4aa3-9858-7acddab9c950","Type":"ContainerDied","Data":"ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8"} Nov 01 08:59:48.106471 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.106310 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7" event={"ID":"53a7f922-e7f1-4aa3-9858-7acddab9c950","Type":"ContainerDied","Data":"52fc04ec75ceab26f00bb32c4dc1471e471b62d0c26c4c417e8881c46fcf0e2c"} Nov 01 08:59:48.117278 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.117253 5710 scope.go:122] "RemoveContainer" containerID="6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5" Nov 01 08:59:48.117626 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:48.117598 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5\": container with ID starting with 6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5 not found: ID does not exist" containerID="6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5" Nov 01 08:59:48.117682 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.117639 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5"} err="failed to get container status \"6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5\": rpc error: code = NotFound desc = could not find container \"6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5\": container with ID starting with 6a26e7381224579331e5a3cd09bfe3b54fd65048de841f1641b3256f1e9415a5 not found: ID does not exist" Nov 01 08:59:48.117682 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.117662 5710 scope.go:122] "RemoveContainer" containerID="ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8" Nov 01 08:59:48.129202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.129178 5710 scope.go:122] "RemoveContainer" containerID="ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8" Nov 01 08:59:48.129568 ip-10-0-143-16 kubenswrapper[5710]: E1101 08:59:48.129550 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8\": container with ID starting with ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8 not found: ID does not exist" containerID="ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8" Nov 01 08:59:48.129613 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.129578 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8"} err="failed to get container status \"ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8\": rpc error: code = NotFound desc = could not find container \"ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8\": container with ID starting with ae2205f71b29e73ab5d5260bdb9feaa800added546d03bb2ecf147d7d50091c8 not found: ID does not exist" Nov 01 08:59:48.267851 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.265475 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62"] Nov 01 08:59:48.270332 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.270302 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fad29-7b849875d5-d9m62"] Nov 01 08:59:48.385755 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.383774 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7"] Nov 01 08:59:48.396393 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:48.395893 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fad29-6c6477b79f-nhvh7"] Nov 01 08:59:49.060675 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:49.060639 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1645573a-e4c5-42f5-8671-9c68931b532c" path="/var/lib/kubelet/pods/1645573a-e4c5-42f5-8671-9c68931b532c/volumes" Nov 01 08:59:49.061295 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:49.061278 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a7f922-e7f1-4aa3-9858-7acddab9c950" path="/var/lib/kubelet/pods/53a7f922-e7f1-4aa3-9858-7acddab9c950/volumes" Nov 01 08:59:54.441743 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.441437 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx"] Nov 01 08:59:54.448404 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.448363 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1645573a-e4c5-42f5-8671-9c68931b532c" containerName="payload-processing" Nov 01 08:59:54.449172 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.449143 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1645573a-e4c5-42f5-8671-9c68931b532c" containerName="payload-processing" Nov 01 08:59:54.449462 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.449446 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3ffff0af-66fc-4b11-9853-60ab0709a496" containerName="maas-api" Nov 01 08:59:54.449663 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.449648 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffff0af-66fc-4b11-9853-60ab0709a496" containerName="maas-api" Nov 01 08:59:54.449926 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.449913 5710 cpu_manager.go:414] "containerMap: removing container" podUID="53a7f922-e7f1-4aa3-9858-7acddab9c950" containerName="payload-pre-processing" Nov 01 08:59:54.450029 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.450018 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a7f922-e7f1-4aa3-9858-7acddab9c950" containerName="payload-pre-processing" Nov 01 08:59:54.450387 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.450365 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a7f922-e7f1-4aa3-9858-7acddab9c950" containerName="payload-pre-processing" Nov 01 08:59:54.450533 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.450521 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1645573a-e4c5-42f5-8671-9c68931b532c" containerName="payload-processing" Nov 01 08:59:54.450640 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.450630 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffff0af-66fc-4b11-9853-60ab0709a496" containerName="maas-api" Nov 01 08:59:54.459784 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.458114 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.536054 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.536013 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d31acd45\"" Nov 01 08:59:54.619747 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.618104 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d31acd45-dockercfg-md55p\"" Nov 01 08:59:54.650745 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.636354 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx"] Nov 01 08:59:54.688752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.663156 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwn6t\" (UniqueName: \"kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.688752 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.663337 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.767937 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.767846 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg"] Nov 01 08:59:54.831756 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.809409 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwn6t\" (UniqueName: \"kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.835979 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.835944 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.837687 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.835108 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:54.839010 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.834999 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg"] Nov 01 08:59:54.839566 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.837329 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.904594 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.898907 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwn6t\" (UniqueName: \"kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t\") pod \"payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:54.948093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.943030 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-sswfg\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:54.948093 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:54.943086 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jdw\" (UniqueName: \"kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-sswfg\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:55.056077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.044914 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-sswfg\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:55.056077 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.044975 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jdw\" (UniqueName: \"kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-sswfg\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:55.061766 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.060312 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-sswfg\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:55.119439 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.112767 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jdw\" (UniqueName: \"kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw\") pod \"payload-processing-e2e-mt-d31acd45-59658f777c-sswfg\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:55.139591 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.139558 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:55.195498 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.195461 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:55.526898 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.526866 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx"] Nov 01 08:59:55.533700 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:59:55.533669 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddef433d_68ce_4ef7_a1f2_772a7db251e1.slice/crio-a1b7375a01416c3e4f9633afe893db2cd37517016a86aa67c9d97c23cb88ef55 WatchSource:0}: Error finding container a1b7375a01416c3e4f9633afe893db2cd37517016a86aa67c9d97c23cb88ef55: Status 404 returned error can't find the container with id a1b7375a01416c3e4f9633afe893db2cd37517016a86aa67c9d97c23cb88ef55 Nov 01 08:59:55.642426 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:55.642390 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg"] Nov 01 08:59:55.648124 ip-10-0-143-16 kubenswrapper[5710]: W1101 08:59:55.648083 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58585a6d_4a10_49dd_ab74_8d0d3cac2fb6.slice/crio-bc063be7f0b6708da45cb64e5e3ac909eed4aef2941c25832e8108e2f650ba18 WatchSource:0}: Error finding container bc063be7f0b6708da45cb64e5e3ac909eed4aef2941c25832e8108e2f650ba18: Status 404 returned error can't find the container with id bc063be7f0b6708da45cb64e5e3ac909eed4aef2941c25832e8108e2f650ba18 Nov 01 08:59:56.277389 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.277339 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" event={"ID":"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6","Type":"ContainerStarted","Data":"c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e"} Nov 01 08:59:56.277615 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.277458 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" event={"ID":"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6","Type":"ContainerStarted","Data":"bc063be7f0b6708da45cb64e5e3ac909eed4aef2941c25832e8108e2f650ba18"} Nov 01 08:59:56.278202 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.278178 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 08:59:56.280000 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.279967 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" event={"ID":"ddef433d-68ce-4ef7-a1f2-772a7db251e1","Type":"ContainerStarted","Data":"845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e"} Nov 01 08:59:56.280136 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.280033 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" event={"ID":"ddef433d-68ce-4ef7-a1f2-772a7db251e1","Type":"ContainerStarted","Data":"a1b7375a01416c3e4f9633afe893db2cd37517016a86aa67c9d97c23cb88ef55"} Nov 01 08:59:56.280617 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.280595 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 08:59:56.388164 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.388107 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" podStartSLOduration=2.388091924 podStartE2EDuration="2.388091924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:59:56.387381866 +0000 UTC m=+1619.475718537" watchObservedRunningTime="2026-11-01 08:59:56.388091924 +0000 UTC m=+1619.476428632" Nov 01 08:59:56.563520 ip-10-0-143-16 kubenswrapper[5710]: I1101 08:59:56.538909 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" podStartSLOduration=2.538887143 podStartE2EDuration="2.538887143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 08:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 08:59:56.528919705 +0000 UTC m=+1619.617256377" watchObservedRunningTime="2026-11-01 08:59:56.538887143 +0000 UTC m=+1619.627223819" Nov 01 09:00:00.163862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.163818 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk"] Nov 01 09:00:00.170512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.170483 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.189126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.189095 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr"] Nov 01 09:00:00.194012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.193980 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc"] Nov 01 09:00:00.194145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.194076 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.198903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.198882 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.210304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.210267 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx"] Nov 01 09:00:00.214913 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.214890 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.226754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.226685 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.226923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.226898 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.227068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.227050 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqml\" (UniqueName: \"kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml\") pod \"maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.227138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.227092 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgdd\" (UniqueName: \"kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.231864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.231846 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\"" Nov 01 09:00:00.263824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.263786 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk"] Nov 01 09:00:00.315508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.315475 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc"] Nov 01 09:00:00.320164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.320132 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr"] Nov 01 09:00:00.323559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.323527 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx"] Nov 01 09:00:00.328755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.328705 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqml\" (UniqueName: \"kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml\") pod \"maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.328897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.328787 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgdd\" (UniqueName: \"kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.328897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.328867 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64fx\" (UniqueName: \"kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx\") pod \"maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.329039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.328917 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.329100 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.329082 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmq7\" (UniqueName: \"kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7\") pod \"maas-api-key-cleanup-29892060-4njkx\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.329154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.329138 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.329344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.329321 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.329435 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.329418 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892060-4njkx\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.329881 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.329860 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.329965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.329949 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.378889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.378843 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv"] Nov 01 09:00:00.384847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.384681 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.395547 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.395519 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqml\" (UniqueName: \"kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml\") pod \"maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.395690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.395650 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgdd\" (UniqueName: \"kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.430688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.430580 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.430688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.430660 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmq7\" (UniqueName: \"kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7\") pod \"maas-api-key-cleanup-29892060-4njkx\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.431046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.430888 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.430946 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.430995 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431023 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431080 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431126 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892060-4njkx\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.431278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431238 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431286 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431324 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64fx\" (UniqueName: \"kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx\") pod \"maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.431425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431370 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.431425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.431399 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.432310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.432287 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.432874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.432851 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892060-4njkx\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.438365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.438337 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c1f453-openshift-default-dockercfg-v8b74\"" Nov 01 09:00:00.441166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.441138 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv"] Nov 01 09:00:00.486154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.486105 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:00.492327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.492296 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmq7\" (UniqueName: \"kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7\") pod \"maas-api-key-cleanup-29892060-4njkx\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.492327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.492318 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64fx\" (UniqueName: \"kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx\") pod \"maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.511059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.511026 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:00.518694 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.518479 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:00.528618 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.528589 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.533526 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.533811 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.533865 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.533904 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.533934 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.533968 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.534071 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.534122 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.534171 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.534609 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.534894 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.535149 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.541064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.537266 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.546085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.542253 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.546085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.545895 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.553697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.551910 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.760908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.755077 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:00.761734 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:00.761270 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln\") pod \"e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:01.038556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.038462 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:01.101342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.101295 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk"] Nov 01 09:00:01.112548 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:01.110784 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e38d39_7004_4bd0_ad44_d47e730f6ede.slice/crio-47a29d93233e6c114de4860f3a865acbb61f7c412c2276617b2e86a28fa5729d WatchSource:0}: Error finding container 47a29d93233e6c114de4860f3a865acbb61f7c412c2276617b2e86a28fa5729d: Status 404 returned error can't find the container with id 47a29d93233e6c114de4860f3a865acbb61f7c412c2276617b2e86a28fa5729d Nov 01 09:00:01.162218 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.162185 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc"] Nov 01 09:00:01.172146 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:01.172100 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc8979d_28df_42f9_afa8_0eea7e1480f9.slice/crio-d794187cb3aa19a1f16c6f6ec1086a844a2da040156a637deacb48bcb2d83cd2 WatchSource:0}: Error finding container d794187cb3aa19a1f16c6f6ec1086a844a2da040156a637deacb48bcb2d83cd2: Status 404 returned error can't find the container with id d794187cb3aa19a1f16c6f6ec1086a844a2da040156a637deacb48bcb2d83cd2 Nov 01 09:00:01.213202 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.213160 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx"] Nov 01 09:00:01.241953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.241915 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr"] Nov 01 09:00:01.283962 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:01.283918 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b1fbb1_e43c_4a8c_aab0_ea33267f0b6c.slice/crio-d7069d07b5e628ba6a97fb4de4cca019328d49e23634a15a4663c1aacf25f441 WatchSource:0}: Error finding container d7069d07b5e628ba6a97fb4de4cca019328d49e23634a15a4663c1aacf25f441: Status 404 returned error can't find the container with id d7069d07b5e628ba6a97fb4de4cca019328d49e23634a15a4663c1aacf25f441 Nov 01 09:00:01.284997 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:01.284968 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aeaeed3_5d35_4925_b03b_f62b7e47f608.slice/crio-febaae83e499e2710bc1dde899a9f96b40508a9ed74a1571967ad55675c83016 WatchSource:0}: Error finding container febaae83e499e2710bc1dde899a9f96b40508a9ed74a1571967ad55675c83016: Status 404 returned error can't find the container with id febaae83e499e2710bc1dde899a9f96b40508a9ed74a1571967ad55675c83016 Nov 01 09:00:01.327582 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.327554 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" event={"ID":"8dc8979d-28df-42f9-afa8-0eea7e1480f9","Type":"ContainerStarted","Data":"d794187cb3aa19a1f16c6f6ec1086a844a2da040156a637deacb48bcb2d83cd2"} Nov 01 09:00:01.328881 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.328857 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerStarted","Data":"febaae83e499e2710bc1dde899a9f96b40508a9ed74a1571967ad55675c83016"} Nov 01 09:00:01.330075 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.330048 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" event={"ID":"c1e38d39-7004-4bd0-ad44-d47e730f6ede","Type":"ContainerStarted","Data":"47a29d93233e6c114de4860f3a865acbb61f7c412c2276617b2e86a28fa5729d"} Nov 01 09:00:01.331607 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.331581 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerStarted","Data":"d7069d07b5e628ba6a97fb4de4cca019328d49e23634a15a4663c1aacf25f441"} Nov 01 09:00:01.433966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.433928 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv"] Nov 01 09:00:01.486028 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:01.485990 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac479b6_2bb8_410e_aca6_d3791e9f8744.slice/crio-2f5ff12aba86d89dfe8955a9e7f74d8379f4036ba4b1c7afa7481de33e5fdc5d WatchSource:0}: Error finding container 2f5ff12aba86d89dfe8955a9e7f74d8379f4036ba4b1c7afa7481de33e5fdc5d: Status 404 returned error can't find the container with id 2f5ff12aba86d89dfe8955a9e7f74d8379f4036ba4b1c7afa7481de33e5fdc5d Nov 01 09:00:01.489658 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.489621 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:00:01.489821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.489694 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:00:01.489821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:01.489744 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:00:02.340802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.340766 5710 generic.go:356] "Generic (PLEG): container finished" podID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerID="a4c9c2f5834ae72e7e9b9262233b50f554df653af75273e076652d1593504dec" exitCode=60 Nov 01 09:00:02.341232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.340845 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerDied","Data":"a4c9c2f5834ae72e7e9b9262233b50f554df653af75273e076652d1593504dec"} Nov 01 09:00:02.341232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.341228 5710 scope.go:122] "RemoveContainer" containerID="a4c9c2f5834ae72e7e9b9262233b50f554df653af75273e076652d1593504dec" Nov 01 09:00:02.342793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.342763 5710 generic.go:356] "Generic (PLEG): container finished" podID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerID="f94f09cf4e0f4927d55f945b9e9864398b31b19ac865535e6a6a465e33b4beed" exitCode=60 Nov 01 09:00:02.342923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.342840 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" event={"ID":"c1e38d39-7004-4bd0-ad44-d47e730f6ede","Type":"ContainerDied","Data":"f94f09cf4e0f4927d55f945b9e9864398b31b19ac865535e6a6a465e33b4beed"} Nov 01 09:00:02.343266 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.343202 5710 scope.go:122] "RemoveContainer" containerID="f94f09cf4e0f4927d55f945b9e9864398b31b19ac865535e6a6a465e33b4beed" Nov 01 09:00:02.344941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.344913 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerStarted","Data":"d5ed27afa07645316ae625fef6a2bafdec036a7319e7199f6defa7ed23a4a17e"} Nov 01 09:00:02.347509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.347485 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" event={"ID":"9ac479b6-2bb8-410e-aca6-d3791e9f8744","Type":"ContainerStarted","Data":"ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd"} Nov 01 09:00:02.347614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.347540 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" event={"ID":"9ac479b6-2bb8-410e-aca6-d3791e9f8744","Type":"ContainerStarted","Data":"2f5ff12aba86d89dfe8955a9e7f74d8379f4036ba4b1c7afa7481de33e5fdc5d"} Nov 01 09:00:02.349573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.349548 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" event={"ID":"8dc8979d-28df-42f9-afa8-0eea7e1480f9","Type":"ContainerStarted","Data":"be07438a37cb021a8b0a53facee233783a10899f615bcc476f99e8185a35da3e"} Nov 01 09:00:02.561770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.561601 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" podStartSLOduration=2.561579895 podStartE2EDuration="2.561579895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:02.559487458 +0000 UTC m=+1625.647824132" watchObservedRunningTime="2026-11-01 09:00:02.561579895 +0000 UTC m=+1625.649916567" Nov 01 09:00:02.676200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.676121 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podStartSLOduration=2.676105708 podStartE2EDuration="2.676105708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:02.672184133 +0000 UTC m=+1625.760520842" watchObservedRunningTime="2026-11-01 09:00:02.676105708 +0000 UTC m=+1625.764442379" Nov 01 09:00:02.922738 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:02.913758 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" podStartSLOduration=2.913737745 podStartE2EDuration="2.913737745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:02.910443197 +0000 UTC m=+1625.998779895" watchObservedRunningTime="2026-11-01 09:00:02.913737745 +0000 UTC m=+1626.002074421" Nov 01 09:00:03.038887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.038841 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:03.045286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.045252 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:03.360347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.360304 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerStarted","Data":"72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff"} Nov 01 09:00:03.362561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.362533 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" event={"ID":"c1e38d39-7004-4bd0-ad44-d47e730f6ede","Type":"ContainerStarted","Data":"84adbb41b828fec0d160aa2b634e22912d1dafa3ab433cc0b9630aa71c754eb2"} Nov 01 09:00:03.364377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.364348 5710 generic.go:356] "Generic (PLEG): container finished" podID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerID="d5ed27afa07645316ae625fef6a2bafdec036a7319e7199f6defa7ed23a4a17e" exitCode=60 Nov 01 09:00:03.364531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.364426 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerDied","Data":"d5ed27afa07645316ae625fef6a2bafdec036a7319e7199f6defa7ed23a4a17e"} Nov 01 09:00:03.364800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.364784 5710 scope.go:122] "RemoveContainer" containerID="d5ed27afa07645316ae625fef6a2bafdec036a7319e7199f6defa7ed23a4a17e" Nov 01 09:00:03.366152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.366127 5710 generic.go:356] "Generic (PLEG): container finished" podID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerID="be07438a37cb021a8b0a53facee233783a10899f615bcc476f99e8185a35da3e" exitCode=60 Nov 01 09:00:03.366258 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.366205 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" event={"ID":"8dc8979d-28df-42f9-afa8-0eea7e1480f9","Type":"ContainerDied","Data":"be07438a37cb021a8b0a53facee233783a10899f615bcc476f99e8185a35da3e"} Nov 01 09:00:03.366558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.366434 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:03.366558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.366452 5710 scope.go:122] "RemoveContainer" containerID="be07438a37cb021a8b0a53facee233783a10899f615bcc476f99e8185a35da3e" Nov 01 09:00:03.367501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.367486 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:00:03.469771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.469669 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" podStartSLOduration=3.469652011 podStartE2EDuration="3.469652011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:03.468249995 +0000 UTC m=+1626.556586672" watchObservedRunningTime="2026-11-01 09:00:03.469652011 +0000 UTC m=+1626.557988683" Nov 01 09:00:03.715112 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:03.715056 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" podStartSLOduration=3.715038819 podStartE2EDuration="3.715038819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:03.69630858 +0000 UTC m=+1626.784645437" watchObservedRunningTime="2026-11-01 09:00:03.715038819 +0000 UTC m=+1626.803375491" Nov 01 09:00:04.377357 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.377252 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" event={"ID":"8dc8979d-28df-42f9-afa8-0eea7e1480f9","Type":"ContainerStarted","Data":"cf209406c0b6d8f633c182b8082cf4eabe5e030a3b57ba4e90d1f8e1f8e9c90c"} Nov 01 09:00:04.379514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.379486 5710 generic.go:356] "Generic (PLEG): container finished" podID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerID="72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff" exitCode=60 Nov 01 09:00:04.379670 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.379573 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerDied","Data":"72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff"} Nov 01 09:00:04.379670 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.379655 5710 scope.go:122] "RemoveContainer" containerID="a4c9c2f5834ae72e7e9b9262233b50f554df653af75273e076652d1593504dec" Nov 01 09:00:04.379952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.379927 5710 scope.go:122] "RemoveContainer" containerID="72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff" Nov 01 09:00:04.380360 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:04.380336 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr_odh-ai-gateway-infra(1aeaeed3-5d35-4925-b03b-f62b7e47f608)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" Nov 01 09:00:04.381697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.381673 5710 generic.go:356] "Generic (PLEG): container finished" podID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerID="84adbb41b828fec0d160aa2b634e22912d1dafa3ab433cc0b9630aa71c754eb2" exitCode=60 Nov 01 09:00:04.381855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.381783 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" event={"ID":"c1e38d39-7004-4bd0-ad44-d47e730f6ede","Type":"ContainerDied","Data":"84adbb41b828fec0d160aa2b634e22912d1dafa3ab433cc0b9630aa71c754eb2"} Nov 01 09:00:04.382160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.382124 5710 scope.go:122] "RemoveContainer" containerID="84adbb41b828fec0d160aa2b634e22912d1dafa3ab433cc0b9630aa71c754eb2" Nov 01 09:00:04.383741 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:04.382808 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk_odh-ai-gateway-infra(c1e38d39-7004-4bd0-ad44-d47e730f6ede)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" Nov 01 09:00:04.385346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.385308 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerStarted","Data":"cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f"} Nov 01 09:00:04.504780 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.484319 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk"] Nov 01 09:00:04.578021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:04.577982 5710 scope.go:122] "RemoveContainer" containerID="f94f09cf4e0f4927d55f945b9e9864398b31b19ac865535e6a6a465e33b4beed" Nov 01 09:00:05.436745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.436283 5710 generic.go:356] "Generic (PLEG): container finished" podID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerID="cf209406c0b6d8f633c182b8082cf4eabe5e030a3b57ba4e90d1f8e1f8e9c90c" exitCode=60 Nov 01 09:00:05.436745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.436500 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" event={"ID":"8dc8979d-28df-42f9-afa8-0eea7e1480f9","Type":"ContainerDied","Data":"cf209406c0b6d8f633c182b8082cf4eabe5e030a3b57ba4e90d1f8e1f8e9c90c"} Nov 01 09:00:05.436745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.436564 5710 scope.go:122] "RemoveContainer" containerID="be07438a37cb021a8b0a53facee233783a10899f615bcc476f99e8185a35da3e" Nov 01 09:00:05.437345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.437295 5710 scope.go:122] "RemoveContainer" containerID="cf209406c0b6d8f633c182b8082cf4eabe5e030a3b57ba4e90d1f8e1f8e9c90c" Nov 01 09:00:05.438214 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:05.437778 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc_odh-ai-gateway-infra(8dc8979d-28df-42f9-afa8-0eea7e1480f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" Nov 01 09:00:05.447756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.446367 5710 scope.go:122] "RemoveContainer" containerID="72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff" Nov 01 09:00:05.447756 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:05.447157 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr_odh-ai-gateway-infra(1aeaeed3-5d35-4925-b03b-f62b7e47f608)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" Nov 01 09:00:05.460744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.450248 5710 scope.go:122] "RemoveContainer" containerID="84adbb41b828fec0d160aa2b634e22912d1dafa3ab433cc0b9630aa71c754eb2" Nov 01 09:00:05.460744 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:05.450649 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk_odh-ai-gateway-infra(c1e38d39-7004-4bd0-ad44-d47e730f6ede)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" Nov 01 09:00:05.488963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.478165 5710 generic.go:356] "Generic (PLEG): container finished" podID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerID="cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f" exitCode=60 Nov 01 09:00:05.488963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.478815 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerDied","Data":"cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f"} Nov 01 09:00:05.488963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.481240 5710 scope.go:122] "RemoveContainer" containerID="cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f" Nov 01 09:00:05.488963 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:05.481744 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892060-4njkx_odh-ai-gateway-infra(96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" Nov 01 09:00:05.539257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.539228 5710 scope.go:122] "RemoveContainer" containerID="d5ed27afa07645316ae625fef6a2bafdec036a7319e7199f6defa7ed23a4a17e" Nov 01 09:00:05.680761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:05.680680 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc"] Nov 01 09:00:06.577184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:06.577142 5710 scope.go:122] "RemoveContainer" containerID="cf209406c0b6d8f633c182b8082cf4eabe5e030a3b57ba4e90d1f8e1f8e9c90c" Nov 01 09:00:06.578283 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:06.578247 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc_odh-ai-gateway-infra(8dc8979d-28df-42f9-afa8-0eea7e1480f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" Nov 01 09:00:06.600755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:06.600706 5710 scope.go:122] "RemoveContainer" containerID="cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f" Nov 01 09:00:06.602538 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:06.601451 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892060-4njkx_odh-ai-gateway-infra(96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" Nov 01 09:00:06.904216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:06.904181 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:07.050478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.050095 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml\" (UniqueName: \"kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml\") pod \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " Nov 01 09:00:07.050478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.050261 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca\") pod \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\" (UID: \"c1e38d39-7004-4bd0-ad44-d47e730f6ede\") " Nov 01 09:00:07.052646 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.052561 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca" pod "c1e38d39-7004-4bd0-ad44-d47e730f6ede" (UID: "c1e38d39-7004-4bd0-ad44-d47e730f6ede"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:07.053474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.053450 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml" pod "c1e38d39-7004-4bd0-ad44-d47e730f6ede" (UID: "c1e38d39-7004-4bd0-ad44-d47e730f6ede"). InnerVolumeSpecName "kube-api-access-4vqml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:07.152863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.152774 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vqml\" (UniqueName: \"kubernetes.io/projected/c1e38d39-7004-4bd0-ad44-d47e730f6ede-kube-api-access-4vqml\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:07.152863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.152805 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e38d39-7004-4bd0-ad44-d47e730f6ede-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:07.291817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.291782 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 09:00:07.292025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.291865 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 09:00:07.592607 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.592570 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" event={"ID":"c1e38d39-7004-4bd0-ad44-d47e730f6ede","Type":"ContainerDied","Data":"47a29d93233e6c114de4860f3a865acbb61f7c412c2276617b2e86a28fa5729d"} Nov 01 09:00:07.593118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.592613 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk" Nov 01 09:00:07.593118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.592649 5710 scope.go:122] "RemoveContainer" containerID="84adbb41b828fec0d160aa2b634e22912d1dafa3ab433cc0b9630aa71c754eb2" Nov 01 09:00:07.910918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.910877 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:07.911207 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.911153 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk"] Nov 01 09:00:07.917136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.917090 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8701a0-29892060-9cqjk"] Nov 01 09:00:07.989473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.989421 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx\" (UniqueName: \"kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx\") pod \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " Nov 01 09:00:07.989688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.989562 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca\") pod \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\" (UID: \"8dc8979d-28df-42f9-afa8-0eea7e1480f9\") " Nov 01 09:00:07.990011 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.989978 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca" pod "8dc8979d-28df-42f9-afa8-0eea7e1480f9" (UID: "8dc8979d-28df-42f9-afa8-0eea7e1480f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:07.990512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.990485 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc8979d-28df-42f9-afa8-0eea7e1480f9-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:07.992090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:07.992062 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx" pod "8dc8979d-28df-42f9-afa8-0eea7e1480f9" (UID: "8dc8979d-28df-42f9-afa8-0eea7e1480f9"). InnerVolumeSpecName "kube-api-access-c64fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:08.095908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:08.093063 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64fx\" (UniqueName: \"kubernetes.io/projected/8dc8979d-28df-42f9-afa8-0eea7e1480f9-kube-api-access-c64fx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:08.601944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:08.601914 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" Nov 01 09:00:08.601944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:08.601933 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc" event={"ID":"8dc8979d-28df-42f9-afa8-0eea7e1480f9","Type":"ContainerDied","Data":"d794187cb3aa19a1f16c6f6ec1086a844a2da040156a637deacb48bcb2d83cd2"} Nov 01 09:00:08.602524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:08.601995 5710 scope.go:122] "RemoveContainer" containerID="cf209406c0b6d8f633c182b8082cf4eabe5e030a3b57ba4e90d1f8e1f8e9c90c" Nov 01 09:00:08.789226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:08.780706 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc"] Nov 01 09:00:08.895820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:08.893329 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5fad29-29892060-hdxjc"] Nov 01 09:00:09.037340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.037299 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn"] Nov 01 09:00:09.039909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.039875 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerName="cleanup" Nov 01 09:00:09.040125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040106 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerName="cleanup" Nov 01 09:00:09.040209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040147 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerName="cleanup" Nov 01 09:00:09.040209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040155 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerName="cleanup" Nov 01 09:00:09.040209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040179 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerName="cleanup" Nov 01 09:00:09.040209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040189 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerName="cleanup" Nov 01 09:00:09.040483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040459 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerName="cleanup" Nov 01 09:00:09.040483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040478 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerName="cleanup" Nov 01 09:00:09.040618 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.040489 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" containerName="cleanup" Nov 01 09:00:09.048099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.048067 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.163666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.163573 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c1f453\"" Nov 01 09:00:09.234489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.234420 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.236553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.236514 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75k7f\" (UniqueName: \"kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f\") pod \"payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.247232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.247174 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc8979d-28df-42f9-afa8-0eea7e1480f9" path="/var/lib/kubelet/pods/8dc8979d-28df-42f9-afa8-0eea7e1480f9/volumes" Nov 01 09:00:09.250818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.250695 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" path="/var/lib/kubelet/pods/c1e38d39-7004-4bd0-ad44-d47e730f6ede/volumes" Nov 01 09:00:09.280752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.262360 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn"] Nov 01 09:00:09.303956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.303923 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l"] Nov 01 09:00:09.346746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.317830 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerName="cleanup" Nov 01 09:00:09.346746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.344049 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerName="cleanup" Nov 01 09:00:09.346746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.344558 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e38d39-7004-4bd0-ad44-d47e730f6ede" containerName="cleanup" Nov 01 09:00:09.355101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.355063 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.355968 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.355939 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.358744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.358655 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75k7f\" (UniqueName: \"kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f\") pod \"payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.381063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.358382 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.428213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.428125 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c1f453-dockercfg-hxcjr\"" Nov 01 09:00:09.521759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.509950 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswjk\" (UniqueName: \"kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk\") pod \"payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.552776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.526200 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.552776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.551550 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75k7f\" (UniqueName: \"kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f\") pod \"payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.552776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.551968 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:09.609914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.609807 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l"] Nov 01 09:00:09.633768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.629284 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.633768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.629446 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswjk\" (UniqueName: \"kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk\") pod \"payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.633768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.630809 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.689633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.689545 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswjk\" (UniqueName: \"kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk\") pod \"payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.866992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.866961 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:09.985727 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:09.985673 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn"] Nov 01 09:00:09.992253 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:09.992223 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ecc876_4aad_480c_b5fa_ac4a2d0aacc2.slice/crio-b5f19140ced1ab141ec9edec16bfd3b324ebce9f40f6209aa604256b07aa9b5b WatchSource:0}: Error finding container b5f19140ced1ab141ec9edec16bfd3b324ebce9f40f6209aa604256b07aa9b5b: Status 404 returned error can't find the container with id b5f19140ced1ab141ec9edec16bfd3b324ebce9f40f6209aa604256b07aa9b5b Nov 01 09:00:10.263831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.263744 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l"] Nov 01 09:00:10.270079 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:10.270043 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8d6382_3aaf_41e9_a3dd_cf3b70a08230.slice/crio-0130343ec05214e939ade6a169ac0749c9554e07177c8c3c689d5f958f22e3e4 WatchSource:0}: Error finding container 0130343ec05214e939ade6a169ac0749c9554e07177c8c3c689d5f958f22e3e4: Status 404 returned error can't find the container with id 0130343ec05214e939ade6a169ac0749c9554e07177c8c3c689d5f958f22e3e4 Nov 01 09:00:10.643928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.643887 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" event={"ID":"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230","Type":"ContainerStarted","Data":"fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc"} Nov 01 09:00:10.644411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.643987 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" event={"ID":"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230","Type":"ContainerStarted","Data":"0130343ec05214e939ade6a169ac0749c9554e07177c8c3c689d5f958f22e3e4"} Nov 01 09:00:10.644411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.644163 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:10.646000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.645968 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" event={"ID":"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2","Type":"ContainerStarted","Data":"4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150"} Nov 01 09:00:10.646153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.646026 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" event={"ID":"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2","Type":"ContainerStarted","Data":"b5f19140ced1ab141ec9edec16bfd3b324ebce9f40f6209aa604256b07aa9b5b"} Nov 01 09:00:10.646153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.646066 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:10.777091 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.777024 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" podStartSLOduration=1.777001297 podStartE2EDuration="1.777001297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:10.767281903 +0000 UTC m=+1633.855618580" watchObservedRunningTime="2026-11-01 09:00:10.777001297 +0000 UTC m=+1633.865337970" Nov 01 09:00:10.943864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:10.941974 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" podStartSLOduration=2.941948947 podStartE2EDuration="2.941948947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:10.894015892 +0000 UTC m=+1633.982352566" watchObservedRunningTime="2026-11-01 09:00:10.941948947 +0000 UTC m=+1634.030285620" Nov 01 09:00:11.351324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.351285 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p"] Nov 01 09:00:11.351529 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.351503 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" containerID="cri-o://7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304" gracePeriod=30 Nov 01 09:00:11.397406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.397363 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l"] Nov 01 09:00:11.397669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.397628 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" containerID="cri-o://6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609" gracePeriod=30 Nov 01 09:00:11.403827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.403792 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz"] Nov 01 09:00:11.404071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.404047 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" containerID="cri-o://bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4" gracePeriod=30 Nov 01 09:00:11.405855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.405829 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6"] Nov 01 09:00:11.406036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:11.406014 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" containerID="cri-o://5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89" gracePeriod=30 Nov 01 09:00:14.051306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.051263 5710 scope.go:122] "RemoveContainer" containerID="72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff" Nov 01 09:00:14.398437 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.398398 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd"] Nov 01 09:00:14.398699 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.398670 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" containerID="cri-o://31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2" gracePeriod=30 Nov 01 09:00:14.555747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.554485 5710 patch_prober.go:29] interesting pod/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:00:14.555747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.554554 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:00:15.003888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.988659 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl"] Nov 01 09:00:15.003888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:14.988957 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" containerID="cri-o://6361519a82cf1721be71d7f0321b217df3425c8dd2410566ac50403680f6dcec" gracePeriod=30 Nov 01 09:00:15.230652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:15.228874 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerStarted","Data":"aea3e1991947707c4aa0d0977e4a81d27a2e2f2a9971be6ccc6e0f8e4730e22d"} Nov 01 09:00:15.801377 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:15.801262 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:00:16.281255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.281222 5710 generic.go:356] "Generic (PLEG): container finished" podID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerID="aea3e1991947707c4aa0d0977e4a81d27a2e2f2a9971be6ccc6e0f8e4730e22d" exitCode=60 Nov 01 09:00:16.281767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.281296 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerDied","Data":"aea3e1991947707c4aa0d0977e4a81d27a2e2f2a9971be6ccc6e0f8e4730e22d"} Nov 01 09:00:16.281767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.281362 5710 scope.go:122] "RemoveContainer" containerID="72ddd94eed97a6b8d969f9909a5c49c4d04364f369973a5fab2c90cfca8d30ff" Nov 01 09:00:16.281920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.281884 5710 scope.go:122] "RemoveContainer" containerID="aea3e1991947707c4aa0d0977e4a81d27a2e2f2a9971be6ccc6e0f8e4730e22d" Nov 01 09:00:16.282328 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:16.282304 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr_odh-ai-gateway-infra(1aeaeed3-5d35-4925-b03b-f62b7e47f608)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" Nov 01 09:00:16.381111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.381054 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr"] Nov 01 09:00:16.788204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.787894 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 09:00:16.812192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.812157 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 09:00:16.823287 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.823248 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 09:00:16.873116 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.873088 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 09:00:16.874240 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874197 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.874240 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874237 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.874941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874287 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.874941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874312 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.874941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874361 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.874941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874416 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.874941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.874445 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.875376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.875352 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.875632 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.875548 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.875859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.875800 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.875859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.875841 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.875969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.875932 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.876030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876017 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876044 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.876155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876113 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876158 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.876213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876195 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.876320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876282 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876314 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.876425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876363 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.876425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876395 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876526 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876432 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876526 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876491 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.876634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876522 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.876634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876553 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.876804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876660 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs\") pod \"63531f67-6d85-41d6-b72f-f067225aca11\" (UID: \"63531f67-6d85-41d6-b72f-f067225aca11\") " Nov 01 09:00:16.876804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876696 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket\") pod \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\" (UID: \"e429971d-6ae7-4d59-9de4-81d00c4bf43b\") " Nov 01 09:00:16.876804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876798 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876980 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876824 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs\") pod \"3c022b90-56fc-4aeb-9a49-27ae002603f6\" (UID: \"3c022b90-56fc-4aeb-9a49-27ae002603f6\") " Nov 01 09:00:16.876980 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.876930 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.877125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877103 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.877279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877261 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.877700 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877672 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.877869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877830 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.877869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877851 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.877869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877865 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.878046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877879 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.878046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877891 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.878046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877907 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.878046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877964 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:16.878046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.877997 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:16.878307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.878187 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.878873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.878850 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.879054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.879036 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.879148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.879131 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.879262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.879244 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.879616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.879590 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:16.879987 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.879963 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.880351 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.880268 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:16.881022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.880993 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "kube-api-access-8thqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.882115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.881986 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "kube-api-access-wtwjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.882439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.882396 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:16.890425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890381 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:16.890425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890380 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.890672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890536 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.890672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890568 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.890672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890666 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.890983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890684 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token" pod "63531f67-6d85-41d6-b72f-f067225aca11" (UID: "63531f67-6d85-41d6-b72f-f067225aca11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.890983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.890875 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy" pod "e429971d-6ae7-4d59-9de4-81d00c4bf43b" (UID: "e429971d-6ae7-4d59-9de4-81d00c4bf43b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.892629 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.892600 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc" pod "3c022b90-56fc-4aeb-9a49-27ae002603f6" (UID: "3c022b90-56fc-4aeb-9a49-27ae002603f6"). InnerVolumeSpecName "kube-api-access-j2fdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.978990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.978946 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979030 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979207 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979285 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979337 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979382 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.979451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979410 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:16.979497 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979475 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979565 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979609 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979668 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket\") pod \"1ec208b0-7f97-4f99-a450-658f6de328bf\" (UID: \"1ec208b0-7f97-4f99-a450-658f6de328bf\") " Nov 01 09:00:16.979857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.979817 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.980066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.980042 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.980181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.980114 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.980970 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.980998 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981015 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981030 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwjq\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-kube-api-access-wtwjq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981044 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981058 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c022b90-56fc-4aeb-9a49-27ae002603f6-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981072 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63531f67-6d85-41d6-b72f-f067225aca11-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981085 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981099 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fdc\" (UniqueName: \"kubernetes.io/projected/3c022b90-56fc-4aeb-9a49-27ae002603f6-kube-api-access-j2fdc\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981114 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981127 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981142 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981157 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thqm\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-kube-api-access-8thqm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981171 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981186 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec208b0-7f97-4f99-a450-658f6de328bf-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981201 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63531f67-6d85-41d6-b72f-f067225aca11-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981214 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981228 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981243 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981258 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981273 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981288 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981302 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e429971d-6ae7-4d59-9de4-81d00c4bf43b-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981315 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c022b90-56fc-4aeb-9a49-27ae002603f6-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981329 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63531f67-6d85-41d6-b72f-f067225aca11-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.981369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981342 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63531f67-6d85-41d6-b72f-f067225aca11-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:16.982608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.981927 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:16.982608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.982397 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "kube-api-access-chswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.982608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.982498 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:16.982904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:16.982888 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo" pod "1ec208b0-7f97-4f99-a450-658f6de328bf" (UID: "1ec208b0-7f97-4f99-a450-658f6de328bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:17.082916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.082586 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:17.082916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.082612 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:17.082916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.082671 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec208b0-7f97-4f99-a450-658f6de328bf-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:17.082916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.082689 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chswd\" (UniqueName: \"kubernetes.io/projected/1ec208b0-7f97-4f99-a450-658f6de328bf-kube-api-access-chswd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:17.292915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.292871 5710 generic.go:356] "Generic (PLEG): container finished" podID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerID="6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609" exitCode=0 Nov 01 09:00:17.293531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.292960 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" Nov 01 09:00:17.293674 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.292959 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" event={"ID":"e429971d-6ae7-4d59-9de4-81d00c4bf43b","Type":"ContainerDied","Data":"6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609"} Nov 01 09:00:17.293844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.293792 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l" event={"ID":"e429971d-6ae7-4d59-9de4-81d00c4bf43b","Type":"ContainerDied","Data":"378d914ca64eaf4c0c4f0b75ba6db9515c2b5e4b6bb76d9ddcf1229188043174"} Nov 01 09:00:17.293844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.293840 5710 scope.go:122] "RemoveContainer" containerID="6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609" Nov 01 09:00:17.300548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.300514 5710 generic.go:356] "Generic (PLEG): container finished" podID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerID="5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89" exitCode=0 Nov 01 09:00:17.300692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.300639 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" event={"ID":"1ec208b0-7f97-4f99-a450-658f6de328bf","Type":"ContainerDied","Data":"5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89"} Nov 01 09:00:17.300760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.300698 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" event={"ID":"1ec208b0-7f97-4f99-a450-658f6de328bf","Type":"ContainerDied","Data":"021e1cab253f2ebe0f9a1be3cf528677860f680914a25c89cba7b87b06731c0a"} Nov 01 09:00:17.300887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.300865 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6" Nov 01 09:00:17.304294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.304262 5710 generic.go:356] "Generic (PLEG): container finished" podID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerID="7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304" exitCode=0 Nov 01 09:00:17.304598 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.304566 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" Nov 01 09:00:17.304813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.304621 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" event={"ID":"3c022b90-56fc-4aeb-9a49-27ae002603f6","Type":"ContainerDied","Data":"7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304"} Nov 01 09:00:17.304813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.304687 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p" event={"ID":"3c022b90-56fc-4aeb-9a49-27ae002603f6","Type":"ContainerDied","Data":"127906acccc491778e03c213ac542bda1facf711ab2278b604e751d1a560161f"} Nov 01 09:00:17.309683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.309663 5710 scope.go:122] "RemoveContainer" containerID="6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609" Nov 01 09:00:17.310504 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:17.310468 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609\": container with ID starting with 6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609 not found: ID does not exist" containerID="6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609" Nov 01 09:00:17.310650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.310511 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609"} err="failed to get container status \"6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609\": rpc error: code = NotFound desc = could not find container \"6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609\": container with ID starting with 6110e50e9d5aea3af1f98d0b18fc71d77cec21d7a08a483ec1cefc9165363609 not found: ID does not exist" Nov 01 09:00:17.310650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.310538 5710 scope.go:122] "RemoveContainer" containerID="5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89" Nov 01 09:00:17.310864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.310841 5710 generic.go:356] "Generic (PLEG): container finished" podID="63531f67-6d85-41d6-b72f-f067225aca11" containerID="bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4" exitCode=0 Nov 01 09:00:17.310923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.310898 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" event={"ID":"63531f67-6d85-41d6-b72f-f067225aca11","Type":"ContainerDied","Data":"bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4"} Nov 01 09:00:17.310971 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.310943 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" event={"ID":"63531f67-6d85-41d6-b72f-f067225aca11","Type":"ContainerDied","Data":"46b999b3c0b0758987d54032b3c4785881d7d4d5eead9120cd3a3468af079779"} Nov 01 09:00:17.311057 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.311040 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz" Nov 01 09:00:17.331216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.331186 5710 scope.go:122] "RemoveContainer" containerID="5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89" Nov 01 09:00:17.331705 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:17.331674 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89\": container with ID starting with 5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89 not found: ID does not exist" containerID="5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89" Nov 01 09:00:17.331822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.331739 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89"} err="failed to get container status \"5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89\": rpc error: code = NotFound desc = could not find container \"5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89\": container with ID starting with 5e25bd16cb280f53621132f70aeba1f53da07c458acdcbd310a8578d78d9da89 not found: ID does not exist" Nov 01 09:00:17.331822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.331767 5710 scope.go:122] "RemoveContainer" containerID="7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304" Nov 01 09:00:17.345586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.345048 5710 scope.go:122] "RemoveContainer" containerID="7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304" Nov 01 09:00:17.345586 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:17.345446 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304\": container with ID starting with 7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304 not found: ID does not exist" containerID="7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304" Nov 01 09:00:17.345586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.345473 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304"} err="failed to get container status \"7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304\": rpc error: code = NotFound desc = could not find container \"7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304\": container with ID starting with 7ece6d45798db8734172a69da33ffb946d642b7e1e8870df4e11737d59d20304 not found: ID does not exist" Nov 01 09:00:17.345586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.345492 5710 scope.go:122] "RemoveContainer" containerID="bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4" Nov 01 09:00:17.362486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.362157 5710 scope.go:122] "RemoveContainer" containerID="bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4" Nov 01 09:00:17.362799 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:17.362693 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4\": container with ID starting with bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4 not found: ID does not exist" containerID="bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4" Nov 01 09:00:17.362799 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.362754 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4"} err="failed to get container status \"bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4\": rpc error: code = NotFound desc = could not find container \"bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4\": container with ID starting with bb114900af81997fcbb2f6ffe9314ee8a8531e9dc30303a3706442f358789ee4 not found: ID does not exist" Nov 01 09:00:17.441285 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.441248 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6"] Nov 01 09:00:17.446013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.445962 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-h54x6"] Nov 01 09:00:17.525380 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.525354 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:17.545305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.545272 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l"] Nov 01 09:00:17.550469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.550435 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lfg8l"] Nov 01 09:00:17.593660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.593622 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca\") pod \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " Nov 01 09:00:17.593903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.593709 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd\" (UniqueName: \"kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd\") pod \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\" (UID: \"1aeaeed3-5d35-4925-b03b-f62b7e47f608\") " Nov 01 09:00:17.594070 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.594044 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca" pod "1aeaeed3-5d35-4925-b03b-f62b7e47f608" (UID: "1aeaeed3-5d35-4925-b03b-f62b7e47f608"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:17.594310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.594295 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeaeed3-5d35-4925-b03b-f62b7e47f608-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:17.596179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.596153 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd" pod "1aeaeed3-5d35-4925-b03b-f62b7e47f608" (UID: "1aeaeed3-5d35-4925-b03b-f62b7e47f608"). InnerVolumeSpecName "kube-api-access-kfgdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:17.659744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.656768 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz"] Nov 01 09:00:17.662550 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.662491 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-zbzcz"] Nov 01 09:00:17.696233 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.696199 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgdd\" (UniqueName: \"kubernetes.io/projected/1aeaeed3-5d35-4925-b03b-f62b7e47f608-kube-api-access-kfgdd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:17.871600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.871489 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p"] Nov 01 09:00:17.878675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:17.878638 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-343207-openshift-default-65c556fd8c-lp46p"] Nov 01 09:00:18.061326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:18.061270 5710 scope.go:122] "RemoveContainer" containerID="cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f" Nov 01 09:00:18.326183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:18.326149 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" Nov 01 09:00:18.326746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:18.326186 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr" event={"ID":"1aeaeed3-5d35-4925-b03b-f62b7e47f608","Type":"ContainerDied","Data":"febaae83e499e2710bc1dde899a9f96b40508a9ed74a1571967ad55675c83016"} Nov 01 09:00:18.326746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:18.326252 5710 scope.go:122] "RemoveContainer" containerID="aea3e1991947707c4aa0d0977e4a81d27a2e2f2a9971be6ccc6e0f8e4730e22d" Nov 01 09:00:18.486426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:18.486369 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr"] Nov 01 09:00:18.491666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:18.491629 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892060-24lnr"] Nov 01 09:00:19.060403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.060371 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" path="/var/lib/kubelet/pods/1aeaeed3-5d35-4925-b03b-f62b7e47f608/volumes" Nov 01 09:00:19.061043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.061025 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" path="/var/lib/kubelet/pods/1ec208b0-7f97-4f99-a450-658f6de328bf/volumes" Nov 01 09:00:19.061557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.061539 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" path="/var/lib/kubelet/pods/3c022b90-56fc-4aeb-9a49-27ae002603f6/volumes" Nov 01 09:00:19.062458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.062440 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63531f67-6d85-41d6-b72f-f067225aca11" path="/var/lib/kubelet/pods/63531f67-6d85-41d6-b72f-f067225aca11/volumes" Nov 01 09:00:19.062954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.062936 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" path="/var/lib/kubelet/pods/e429971d-6ae7-4d59-9de4-81d00c4bf43b/volumes" Nov 01 09:00:19.340154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.340130 5710 generic.go:356] "Generic (PLEG): container finished" podID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerID="5877694fb3bd2a7f1eb510a41b2ac230ce2be26ad959596ac64d3f04dd4c7b06" exitCode=60 Nov 01 09:00:19.340577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.340172 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerDied","Data":"5877694fb3bd2a7f1eb510a41b2ac230ce2be26ad959596ac64d3f04dd4c7b06"} Nov 01 09:00:19.340577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.340236 5710 scope.go:122] "RemoveContainer" containerID="cb21c98851dd1481d9f05106d9087c82b4a5687552cb02e2a85e3107a5afc97f" Nov 01 09:00:19.340802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.340781 5710 scope.go:122] "RemoveContainer" containerID="5877694fb3bd2a7f1eb510a41b2ac230ce2be26ad959596ac64d3f04dd4c7b06" Nov 01 09:00:19.341233 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:19.341209 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892060-4njkx_odh-ai-gateway-infra(96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" Nov 01 09:00:19.695560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.695532 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 09:00:19.736047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736007 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736078 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736107 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736140 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736169 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736398 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736276 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736398 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736374 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:19.736501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736451 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:19.736501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736474 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736519 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736589 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert\") pod \"612f4800-78b3-49ff-9b00-41415daa0fea\" (UID: \"612f4800-78b3-49ff-9b00-41415daa0fea\") " Nov 01 09:00:19.736791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736744 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:19.736867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.736811 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:19.737184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.737159 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:19.737254 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.737237 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.737318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.737258 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.737318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.737274 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.737318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.737287 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.738573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.738548 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:19.738731 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.738696 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:19.739053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.739037 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:19.739250 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.739235 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp" pod "612f4800-78b3-49ff-9b00-41415daa0fea" (UID: "612f4800-78b3-49ff-9b00-41415daa0fea"). InnerVolumeSpecName "kube-api-access-9dkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:19.838943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.838906 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612f4800-78b3-49ff-9b00-41415daa0fea-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.838943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.838938 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612f4800-78b3-49ff-9b00-41415daa0fea-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.838943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.838949 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.839286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.838960 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkkp\" (UniqueName: \"kubernetes.io/projected/612f4800-78b3-49ff-9b00-41415daa0fea-kube-api-access-9dkkp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:19.839286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:19.838973 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612f4800-78b3-49ff-9b00-41415daa0fea-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.350973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.350936 5710 generic.go:356] "Generic (PLEG): container finished" podID="612f4800-78b3-49ff-9b00-41415daa0fea" containerID="31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2" exitCode=0 Nov 01 09:00:20.351558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.351021 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" event={"ID":"612f4800-78b3-49ff-9b00-41415daa0fea","Type":"ContainerDied","Data":"31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2"} Nov 01 09:00:20.351558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.351042 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" Nov 01 09:00:20.351558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.351104 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd" event={"ID":"612f4800-78b3-49ff-9b00-41415daa0fea","Type":"ContainerDied","Data":"ce794b9cadde389379eb2e5caade500cfacf0cd20c30d179f0169b3af8dfd988"} Nov 01 09:00:20.351558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.351136 5710 scope.go:122] "RemoveContainer" containerID="31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2" Nov 01 09:00:20.356626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.356596 5710 generic.go:356] "Generic (PLEG): container finished" podID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerID="6361519a82cf1721be71d7f0321b217df3425c8dd2410566ac50403680f6dcec" exitCode=0 Nov 01 09:00:20.356827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.356747 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" event={"ID":"34ea95b4-423a-4c19-8437-45dc72aa2ba5","Type":"ContainerDied","Data":"6361519a82cf1721be71d7f0321b217df3425c8dd2410566ac50403680f6dcec"} Nov 01 09:00:20.364361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.364335 5710 scope.go:122] "RemoveContainer" containerID="31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2" Nov 01 09:00:20.364760 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:20.364738 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2\": container with ID starting with 31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2 not found: ID does not exist" containerID="31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2" Nov 01 09:00:20.364841 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.364769 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2"} err="failed to get container status \"31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2\": rpc error: code = NotFound desc = could not find container \"31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2\": container with ID starting with 31f5e8eb4becb7dcd75bfd2f2e2579311d260c33166c28d3f38170ac8e151ef2 not found: ID does not exist" Nov 01 09:00:20.364841 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.364787 5710 scope.go:122] "RemoveContainer" containerID="d8c96d870070fecfda7fc3fe5223ab40ee22a1f0eababd0821a61fac5c0c6195" Nov 01 09:00:20.469276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.469063 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx"] Nov 01 09:00:20.556658 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.556609 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd"] Nov 01 09:00:20.562558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.562527 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-7nfdd"] Nov 01 09:00:20.657602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.657566 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:20.761575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.761547 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 09:00:20.761811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.761793 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.761899 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.761859 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.761990 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762057 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762032 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7\" (UniqueName: \"kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7\") pod \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " Nov 01 09:00:20.762112 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762075 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762119 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762131 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:20.762272 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762212 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca\") pod \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\" (UID: \"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c\") " Nov 01 09:00:20.762272 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762259 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762287 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762280 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:20.762370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762349 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762351 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:20.762561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762451 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token\") pod \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\" (UID: \"34ea95b4-423a-4c19-8437-45dc72aa2ba5\") " Nov 01 09:00:20.762561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762539 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:20.762653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762604 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca" pod "96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" (UID: "96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:20.762957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.762929 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:20.763134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.763083 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.763257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.763143 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.763257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.763156 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.763257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.763176 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.763257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.763189 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.763257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.763201 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.764828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.764803 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:20.764934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.764839 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "kube-api-access-q9w5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:20.764993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.764938 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:20.764993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.764989 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7" pod "96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" (UID: "96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c"). InnerVolumeSpecName "kube-api-access-9cmq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:20.765098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.765083 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo" pod "34ea95b4-423a-4c19-8437-45dc72aa2ba5" (UID: "34ea95b4-423a-4c19-8437-45dc72aa2ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:20.864544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.864460 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmq7\" (UniqueName: \"kubernetes.io/projected/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c-kube-api-access-9cmq7\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.864544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.864493 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.864544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.864504 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9w5c\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-kube-api-access-q9w5c\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.864544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.864513 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:20.864544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:20.864522 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ea95b4-423a-4c19-8437-45dc72aa2ba5-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:21.060427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.060391 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" path="/var/lib/kubelet/pods/612f4800-78b3-49ff-9b00-41415daa0fea/volumes" Nov 01 09:00:21.395283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.395246 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" Nov 01 09:00:21.395820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.395242 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx" event={"ID":"96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c","Type":"ContainerDied","Data":"d7069d07b5e628ba6a97fb4de4cca019328d49e23634a15a4663c1aacf25f441"} Nov 01 09:00:21.395820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.395399 5710 scope.go:122] "RemoveContainer" containerID="5877694fb3bd2a7f1eb510a41b2ac230ce2be26ad959596ac64d3f04dd4c7b06" Nov 01 09:00:21.398107 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.397809 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" event={"ID":"34ea95b4-423a-4c19-8437-45dc72aa2ba5","Type":"ContainerDied","Data":"f90dd2bb04eb830845d45d4b7170e3935303ea2322c36da5cfd455e4b2477f50"} Nov 01 09:00:21.398107 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.397940 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl" Nov 01 09:00:21.411793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.411765 5710 scope.go:122] "RemoveContainer" containerID="6361519a82cf1721be71d7f0321b217df3425c8dd2410566ac50403680f6dcec" Nov 01 09:00:21.554188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.554157 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx"] Nov 01 09:00:21.568783 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.568751 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892060-4njkx"] Nov 01 09:00:21.657488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.657410 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:00:21.657764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.657748 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:00:21.702851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.702817 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl"] Nov 01 09:00:21.709271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:21.709240 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5fad29-openshift-default-5877bd5c95-27xxl"] Nov 01 09:00:22.011276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:22.011181 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn"] Nov 01 09:00:22.134211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:22.134168 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx"] Nov 01 09:00:22.174373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:22.174337 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8"] Nov 01 09:00:22.174583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:22.174557 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" containerID="cri-o://38f999c9c319caaaadc01db4b2469dbfd77e6297ac3143054a6fc7e41ddf4633" gracePeriod=30 Nov 01 09:00:23.061068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:23.061036 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" path="/var/lib/kubelet/pods/34ea95b4-423a-4c19-8437-45dc72aa2ba5/volumes" Nov 01 09:00:23.061520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:23.061505 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" path="/var/lib/kubelet/pods/96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c/volumes" Nov 01 09:00:27.012169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.012120 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" podUID="b8c57c63-9834-43b2-a18b-adff7d82abc0" containerName="payload-processing" containerID="cri-o://281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb" gracePeriod=25 Nov 01 09:00:27.135381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.135338 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" podUID="8a941e09-bdf9-451a-be9b-1c9db5bba1f0" containerName="payload-pre-processing" containerID="cri-o://e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1" gracePeriod=25 Nov 01 09:00:27.320019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.319987 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 09:00:27.326898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.326859 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 09:00:27.327062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.326922 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 09:00:27.418392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.418343 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume\") pod \"b8c57c63-9834-43b2-a18b-adff7d82abc0\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " Nov 01 09:00:27.418591 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.418568 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b\" (UniqueName: \"kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b\") pod \"b8c57c63-9834-43b2-a18b-adff7d82abc0\" (UID: \"b8c57c63-9834-43b2-a18b-adff7d82abc0\") " Nov 01 09:00:27.418871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.418837 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume" pod "b8c57c63-9834-43b2-a18b-adff7d82abc0" (UID: "b8c57c63-9834-43b2-a18b-adff7d82abc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:27.419336 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.419309 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c57c63-9834-43b2-a18b-adff7d82abc0-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.422791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.422758 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b" pod "b8c57c63-9834-43b2-a18b-adff7d82abc0" (UID: "b8c57c63-9834-43b2-a18b-adff7d82abc0"). InnerVolumeSpecName "kube-api-access-sr46b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:27.465386 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.465355 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 09:00:27.481170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.480992 5710 generic.go:356] "Generic (PLEG): container finished" podID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerID="38f999c9c319caaaadc01db4b2469dbfd77e6297ac3143054a6fc7e41ddf4633" exitCode=0 Nov 01 09:00:27.481170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.481120 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" event={"ID":"7259d009-0b44-423b-b235-2ded03c6b9e0","Type":"ContainerDied","Data":"38f999c9c319caaaadc01db4b2469dbfd77e6297ac3143054a6fc7e41ddf4633"} Nov 01 09:00:27.482958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.482925 5710 generic.go:356] "Generic (PLEG): container finished" podID="b8c57c63-9834-43b2-a18b-adff7d82abc0" containerID="281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb" exitCode=0 Nov 01 09:00:27.483135 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.483117 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" Nov 01 09:00:27.484401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.484373 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" event={"ID":"b8c57c63-9834-43b2-a18b-adff7d82abc0","Type":"ContainerDied","Data":"281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb"} Nov 01 09:00:27.484608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.484586 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn" event={"ID":"b8c57c63-9834-43b2-a18b-adff7d82abc0","Type":"ContainerDied","Data":"4040893ad29411b16ca37b18ce15622b2bccd6689ff34704238f1729a36611f8"} Nov 01 09:00:27.484858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.484840 5710 scope.go:122] "RemoveContainer" containerID="281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb" Nov 01 09:00:27.487967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.487941 5710 generic.go:356] "Generic (PLEG): container finished" podID="8a941e09-bdf9-451a-be9b-1c9db5bba1f0" containerID="e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1" exitCode=0 Nov 01 09:00:27.488222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.488202 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" event={"ID":"8a941e09-bdf9-451a-be9b-1c9db5bba1f0","Type":"ContainerDied","Data":"e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1"} Nov 01 09:00:27.488368 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.488349 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" event={"ID":"8a941e09-bdf9-451a-be9b-1c9db5bba1f0","Type":"ContainerDied","Data":"56ea6433d22daae5ee18897e6d50c040dd4f1d0ad84809c84b3edcb2b4f9a87e"} Nov 01 09:00:27.488542 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.488525 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx" Nov 01 09:00:27.502645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.502607 5710 scope.go:122] "RemoveContainer" containerID="281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb" Nov 01 09:00:27.503110 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:27.503076 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb\": container with ID starting with 281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb not found: ID does not exist" containerID="281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb" Nov 01 09:00:27.503216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.503125 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb"} err="failed to get container status \"281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb\": rpc error: code = NotFound desc = could not find container \"281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb\": container with ID starting with 281d90037f668c68730df25b3b9201bcffa9459b38216aec6c976bd317207edb not found: ID does not exist" Nov 01 09:00:27.503216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.503170 5710 scope.go:122] "RemoveContainer" containerID="e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1" Nov 01 09:00:27.515101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.515078 5710 scope.go:122] "RemoveContainer" containerID="e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1" Nov 01 09:00:27.515546 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:27.515515 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1\": container with ID starting with e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1 not found: ID does not exist" containerID="e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1" Nov 01 09:00:27.515661 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.515553 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1"} err="failed to get container status \"e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1\": rpc error: code = NotFound desc = could not find container \"e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1\": container with ID starting with e6ca4cc10cfa75009fbb4cf24caefe8050d0f3256c9deea5679f4d85fc2d12d1 not found: ID does not exist" Nov 01 09:00:27.521014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.520985 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 09:00:27.521239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.521219 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr46b\" (UniqueName: \"kubernetes.io/projected/b8c57c63-9834-43b2-a18b-adff7d82abc0-kube-api-access-sr46b\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.694209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.694172 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.694504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.694489 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.694647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.694633 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.694811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.694796 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.694928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.694916 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume\") pod \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " Nov 01 09:00:27.695119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.695023 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5\" (UniqueName: \"kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5\") pod \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\" (UID: \"8a941e09-bdf9-451a-be9b-1c9db5bba1f0\") " Nov 01 09:00:27.695275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.695259 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.695384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.695373 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.695484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.695473 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.695574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.695564 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.697197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.697168 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert\") pod \"7259d009-0b44-423b-b235-2ded03c6b9e0\" (UID: \"7259d009-0b44-423b-b235-2ded03c6b9e0\") " Nov 01 09:00:27.709932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.709883 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:27.710470 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.710437 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:27.710844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.710824 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:27.775749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.770823 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume" pod "8a941e09-bdf9-451a-be9b-1c9db5bba1f0" (UID: "8a941e09-bdf9-451a-be9b-1c9db5bba1f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:27.793450 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.792875 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:27.795125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.795093 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn"] Nov 01 09:00:27.804781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.800822 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.804781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.800861 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.804781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.800878 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.804781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.800893 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.804781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.800907 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.821684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.821632 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:27.828265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.828221 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d0ba8f-7d568f46dc-jvnbn"] Nov 01 09:00:27.832680 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.830568 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:27.832948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.830834 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "kube-api-access-ptwbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:27.833086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.832271 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5" pod "8a941e09-bdf9-451a-be9b-1c9db5bba1f0" (UID: "8a941e09-bdf9-451a-be9b-1c9db5bba1f0"). InnerVolumeSpecName "kube-api-access-tzzp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:27.833192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.832476 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:27.833302 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.832563 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo" pod "7259d009-0b44-423b-b235-2ded03c6b9e0" (UID: "7259d009-0b44-423b-b235-2ded03c6b9e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:27.903032 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.902986 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.903032 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.903031 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.903265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.903050 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwbg\" (UniqueName: \"kubernetes.io/projected/7259d009-0b44-423b-b235-2ded03c6b9e0-kube-api-access-ptwbg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.903265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.903067 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7259d009-0b44-423b-b235-2ded03c6b9e0-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.903265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.903081 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzzp5\" (UniqueName: \"kubernetes.io/projected/8a941e09-bdf9-451a-be9b-1c9db5bba1f0-kube-api-access-tzzp5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:27.903265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:27.903098 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7259d009-0b44-423b-b235-2ded03c6b9e0-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:28.203810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.203769 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx"] Nov 01 09:00:28.250782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.250712 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d0ba8f-5455cf499-jbjvx"] Nov 01 09:00:28.497342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.497311 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" Nov 01 09:00:28.497521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.497307 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8" event={"ID":"7259d009-0b44-423b-b235-2ded03c6b9e0","Type":"ContainerDied","Data":"debc48829a1320207374743ccc25f72c8fa04fd75f84b7d130a99e97ed2e3c66"} Nov 01 09:00:28.497521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.497423 5710 scope.go:122] "RemoveContainer" containerID="38f999c9c319caaaadc01db4b2469dbfd77e6297ac3143054a6fc7e41ddf4633" Nov 01 09:00:28.661544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.661508 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8"] Nov 01 09:00:28.708404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:28.708365 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-m4ms8"] Nov 01 09:00:29.060495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.060454 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" path="/var/lib/kubelet/pods/7259d009-0b44-423b-b235-2ded03c6b9e0/volumes" Nov 01 09:00:29.061038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.061023 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a941e09-bdf9-451a-be9b-1c9db5bba1f0" path="/var/lib/kubelet/pods/8a941e09-bdf9-451a-be9b-1c9db5bba1f0/volumes" Nov 01 09:00:29.061439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.061426 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c57c63-9834-43b2-a18b-adff7d82abc0" path="/var/lib/kubelet/pods/b8c57c63-9834-43b2-a18b-adff7d82abc0/volumes" Nov 01 09:00:29.500489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.500447 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8"] Nov 01 09:00:29.501157 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.500843 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" containerID="cri-o://b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2" gracePeriod=30 Nov 01 09:00:29.547549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.547502 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc"] Nov 01 09:00:29.548037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.548000 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" containerID="cri-o://332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046" gracePeriod=30 Nov 01 09:00:29.551066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.551042 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b"] Nov 01 09:00:29.551298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.551271 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" containerID="cri-o://5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d" gracePeriod=30 Nov 01 09:00:29.595292 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.595250 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd"] Nov 01 09:00:29.595568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.595502 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" containerID="cri-o://4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81" gracePeriod=30 Nov 01 09:00:29.598543 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.598509 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2"] Nov 01 09:00:29.598819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:29.598787 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" containerID="cri-o://08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158" gracePeriod=30 Nov 01 09:00:33.129145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129098 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr"] Nov 01 09:00:33.129934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129913 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" Nov 01 09:00:33.129934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129936 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129945 5710 cpu_manager.go:414] "containerMap: removing container" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129950 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129960 5710 cpu_manager.go:414] "containerMap: removing container" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129965 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129980 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.129985 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130004 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130008 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130015 5710 cpu_manager.go:414] "containerMap: removing container" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130019 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130025 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b8c57c63-9834-43b2-a18b-adff7d82abc0" containerName="payload-processing" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130030 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c57c63-9834-43b2-a18b-adff7d82abc0" containerName="payload-processing" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130038 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130042 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130050 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130054 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130060 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130064 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130070 5710 cpu_manager.go:414] "containerMap: removing container" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130074 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130083 5710 cpu_manager.go:414] "containerMap: removing container" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130089 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130099 5710 cpu_manager.go:414] "containerMap: removing container" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130103 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130114 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8a941e09-bdf9-451a-be9b-1c9db5bba1f0" containerName="payload-pre-processing" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130118 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a941e09-bdf9-451a-be9b-1c9db5bba1f0" containerName="payload-pre-processing" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130135 5710 cpu_manager.go:414] "containerMap: removing container" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130140 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130234 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec208b0-7f97-4f99-a450-658f6de328bf" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130242 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130247 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c57c63-9834-43b2-a18b-adff7d82abc0" containerName="payload-processing" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130254 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130262 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130266 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeaeed3-5d35-4925-b03b-f62b7e47f608" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130274 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="63531f67-6d85-41d6-b72f-f067225aca11" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130280 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130285 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a941e09-bdf9-451a-be9b-1c9db5bba1f0" containerName="payload-pre-processing" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130291 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="7259d009-0b44-423b-b235-2ded03c6b9e0" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130295 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130301 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e429971d-6ae7-4d59-9de4-81d00c4bf43b" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130306 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130312 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="612f4800-78b3-49ff-9b00-41415daa0fea" containerName="istio-proxy" Nov 01 09:00:33.130619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.130319 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c022b90-56fc-4aeb-9a49-27ae002603f6" containerName="istio-proxy" Nov 01 09:00:33.136711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.136688 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.184500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.184444 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.184773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.184552 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.184773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.184674 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.185155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.184853 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmvg\" (UniqueName: \"kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.188654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.188628 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr"] Nov 01 09:00:33.231514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.231477 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d31acd45-serving-cert\"" Nov 01 09:00:33.286705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.286569 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmvg\" (UniqueName: \"kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.287193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.286789 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.287193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.286827 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.287193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.286943 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.287495 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:33.287471 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-serving-cert: secret "maas-api-e2e-mt-d31acd45-serving-cert" not found Nov 01 09:00:33.287685 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:33.287564 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls podName:a1471ad3-a93f-4a83-bd17-5e3008d2cf2b nodeName:}" failed. No retries permitted until 2026-11-01 09:00:33.787540872 +0000 UTC m=+1656.875877524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls") pod "maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" (UID: "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b") : secret "maas-api-e2e-mt-d31acd45-serving-cert" not found Nov 01 09:00:33.287896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.287872 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.290303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.290282 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.345904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.345867 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmvg\" (UniqueName: \"kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.805371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.805332 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.808174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.808141 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\") pod \"maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:33.845432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.845396 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:00:33.845614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:33.845457 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:00:34.051545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.051510 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:34.446056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.446022 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr"] Nov 01 09:00:34.452871 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:34.452842 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1471ad3_a93f_4a83_bd17_5e3008d2cf2b.slice/crio-fb6a37047b472d5d8e301ca28bc8e16d0ab8fffe9cec6ddcd1114fd2815e97f3 WatchSource:0}: Error finding container fb6a37047b472d5d8e301ca28bc8e16d0ab8fffe9cec6ddcd1114fd2815e97f3: Status 404 returned error can't find the container with id fb6a37047b472d5d8e301ca28bc8e16d0ab8fffe9cec6ddcd1114fd2815e97f3 Nov 01 09:00:34.454391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.454370 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:00:34.604877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.604832 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" event={"ID":"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b","Type":"ContainerStarted","Data":"fb6a37047b472d5d8e301ca28bc8e16d0ab8fffe9cec6ddcd1114fd2815e97f3"} Nov 01 09:00:34.839784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.839029 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj"] Nov 01 09:00:34.842683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.840515 5710 cpu_manager.go:414] "containerMap: removing container" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" Nov 01 09:00:34.842683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.840552 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ea95b4-423a-4c19-8437-45dc72aa2ba5" containerName="istio-proxy" Nov 01 09:00:34.842683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.840839 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b1fbb1-e43c-4a8c-aab0-ea33267f0b6c" containerName="cleanup" Nov 01 09:00:34.865085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.865054 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.883127 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.877402 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj"] Nov 01 09:00:34.917674 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.917641 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-568f63-openshift-default-dockercfg-nbmgq\"" Nov 01 09:00:34.946798 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.946759 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.946925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.946812 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.946925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.946859 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.946925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.946906 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.947321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.946932 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.947321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.946964 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.947321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.947143 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5z4f\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.947321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.947209 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:34.947594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:34.947344 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.090923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.090874 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.091327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.091284 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.091499 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.091483 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.091675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.091635 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.104264 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.104220 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.111917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.111884 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.112198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.112175 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.112612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.112589 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z4f\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.112789 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.112775 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.113415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.113393 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.113538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.103043 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.113940 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.092163 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.114603 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.103246 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.114603 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.111534 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.124847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.124763 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.199627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.199592 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.201958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.201891 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.203181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.203085 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z4f\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f\") pod \"e2e-api-b-568f63-openshift-default-59f578746c-4zwrj\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.215602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.215577 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:35.217467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.217179 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 09:00:35.395796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.395770 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 09:00:35.398351 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.398330 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 09:00:35.425362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425326 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425477 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425529 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425584 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425604 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425657 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425680 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425709 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.425873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.425813 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data\") pod \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\" (UID: \"c0d956d1-cc92-4492-98a0-6e485dc9fd0d\") " Nov 01 09:00:35.426581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.426555 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.426732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.426704 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.427415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.427109 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:35.431613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.429115 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.432031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.432007 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.435756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.435672 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.436020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.435981 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.437032 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.436431 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:35.437843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.437790 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt" pod "c0d956d1-cc92-4492-98a0-6e485dc9fd0d" (UID: "c0d956d1-cc92-4492-98a0-6e485dc9fd0d"). InnerVolumeSpecName "kube-api-access-7g2zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.440168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.440142 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545446 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545622 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545683 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545761 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545798 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545907 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.545943 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.546019 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.546055 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.547845 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.547938 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.547986 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548029 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548116 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548157 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548234 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548264 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548311 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548382 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548421 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548464 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548503 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548565 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548597 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket\") pod \"4d595622-6926-49df-b3f9-682fa21c727d\" (UID: \"4d595622-6926-49df-b3f9-682fa21c727d\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548657 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk\") pod \"a8e47564-b224-41c0-b438-6c00d3dec913\" (UID: \"a8e47564-b224-41c0-b438-6c00d3dec913\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548689 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.548742 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs\") pod \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\" (UID: \"82b6f90f-bf37-48be-aeba-1e6ef73b373b\") " Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549514 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549534 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549548 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549563 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549577 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549590 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549606 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549620 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2zt\" (UniqueName: \"kubernetes.io/projected/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-kube-api-access-7g2zt\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.559975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.549637 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d956d1-cc92-4492-98a0-6e485dc9fd0d-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.581872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.569901 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 09:00:35.581872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.546682 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:35.581872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.570036 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.581872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.581843 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.583027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.582593 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.583370 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.585908 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.586678 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.589807 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "kube-api-access-rcsqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.590258 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.590437 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.590552 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.590756 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:35.592912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.591206 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.596584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.593673 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.596584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.594588 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.596584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.594905 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.597500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.597237 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.598799 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.598756 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.598892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.598837 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.600379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.599078 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.604636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.604595 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.605306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.605278 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "kube-api-access-nv9v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.605443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.605414 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86" pod "82b6f90f-bf37-48be-aeba-1e6ef73b373b" (UID: "82b6f90f-bf37-48be-aeba-1e6ef73b373b"). InnerVolumeSpecName "kube-api-access-vbm86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.605684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.605547 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.605951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.605846 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:35.605951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.605874 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token" pod "a8e47564-b224-41c0-b438-6c00d3dec913" (UID: "a8e47564-b224-41c0-b438-6c00d3dec913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.607849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.607819 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo" pod "4d595622-6926-49df-b3f9-682fa21c727d" (UID: "4d595622-6926-49df-b3f9-682fa21c727d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:35.623737 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.623544 5710 generic.go:356] "Generic (PLEG): container finished" podID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerID="b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2" exitCode=0 Nov 01 09:00:35.623737 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.623608 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" Nov 01 09:00:35.623737 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.623623 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" event={"ID":"c0d956d1-cc92-4492-98a0-6e485dc9fd0d","Type":"ContainerDied","Data":"b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2"} Nov 01 09:00:35.623737 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.623682 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8" event={"ID":"c0d956d1-cc92-4492-98a0-6e485dc9fd0d","Type":"ContainerDied","Data":"977b2c37299fde268a667b12da7ab9d48521ad801290a0ba4dde89768715c045"} Nov 01 09:00:35.624015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.623747 5710 scope.go:122] "RemoveContainer" containerID="b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2" Nov 01 09:00:35.626208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.626041 5710 generic.go:356] "Generic (PLEG): container finished" podID="a8e47564-b224-41c0-b438-6c00d3dec913" containerID="5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d" exitCode=0 Nov 01 09:00:35.626208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.626089 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" Nov 01 09:00:35.626208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.626129 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" event={"ID":"a8e47564-b224-41c0-b438-6c00d3dec913","Type":"ContainerDied","Data":"5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d"} Nov 01 09:00:35.626208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.626169 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b" event={"ID":"a8e47564-b224-41c0-b438-6c00d3dec913","Type":"ContainerDied","Data":"a819528a4428d19d4d493b7bb5cebca46a836812dd8b142bd64084fea663c23e"} Nov 01 09:00:35.635109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.632341 5710 generic.go:356] "Generic (PLEG): container finished" podID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerID="08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158" exitCode=0 Nov 01 09:00:35.635109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.632489 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" Nov 01 09:00:35.635109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.634819 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" event={"ID":"82b6f90f-bf37-48be-aeba-1e6ef73b373b","Type":"ContainerDied","Data":"08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158"} Nov 01 09:00:35.635109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.634880 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2" event={"ID":"82b6f90f-bf37-48be-aeba-1e6ef73b373b","Type":"ContainerDied","Data":"b24db8bccc276bd87d3a2b5759c7416d045fbc4257c69597d51f7ca37619d09e"} Nov 01 09:00:35.636671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.636431 5710 generic.go:356] "Generic (PLEG): container finished" podID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerID="4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81" exitCode=0 Nov 01 09:00:35.636671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.636502 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" Nov 01 09:00:35.636671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.636600 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" event={"ID":"a2bacc7b-11be-442c-b337-815ac913ffe3","Type":"ContainerDied","Data":"4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81"} Nov 01 09:00:35.636671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.636639 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd" event={"ID":"a2bacc7b-11be-442c-b337-815ac913ffe3","Type":"ContainerDied","Data":"21cc09ea209a9b1df962e79a01e46b2ed8d5f18fe78b44b621b05ba9e370dd1a"} Nov 01 09:00:35.638663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.638637 5710 generic.go:356] "Generic (PLEG): container finished" podID="4d595622-6926-49df-b3f9-682fa21c727d" containerID="332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046" exitCode=0 Nov 01 09:00:35.638815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.638709 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" event={"ID":"4d595622-6926-49df-b3f9-682fa21c727d","Type":"ContainerDied","Data":"332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046"} Nov 01 09:00:35.638815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.638772 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" event={"ID":"4d595622-6926-49df-b3f9-682fa21c727d","Type":"ContainerDied","Data":"04a1828429c1709b76da2443331ed1f0b92bc480f2526edf326a8df791b5e659"} Nov 01 09:00:35.638933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.638846 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc" Nov 01 09:00:35.653185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.653135 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655105 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655085 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655351 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655470 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.655637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655625 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655690 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655827 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655920 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.655962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655950 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.656241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.655978 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x\") pod \"a2bacc7b-11be-442c-b337-815ac913ffe3\" (UID: \"a2bacc7b-11be-442c-b337-815ac913ffe3\") " Nov 01 09:00:35.656524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.656502 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:00:35.656633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.656605 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.656802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.656782 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.656901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.656878 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.657005 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.656986 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:00:35.657269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.657251 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.657375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.657362 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e47564-b224-41c0-b438-6c00d3dec913-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.657468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.657455 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658120 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.658279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658265 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658368 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658357 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658447 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658436 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658581 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658680 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658783 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcsqk\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-kube-api-access-rcsqk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.658879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.658868 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.659776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.659759 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.660149 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.660131 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.661331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.661315 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2bacc7b-11be-442c-b337-815ac913ffe3-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.661571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.661555 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e47564-b224-41c0-b438-6c00d3dec913-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.661686 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.661673 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.661829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.661818 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.661990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.661974 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d595622-6926-49df-b3f9-682fa21c727d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662086 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e47564-b224-41c0-b438-6c00d3dec913-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662279 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662357 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662416 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662408 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662465 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662456 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b6f90f-bf37-48be-aeba-1e6ef73b373b-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662512 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662563 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662611 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbm86\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-kube-api-access-vbm86\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662658 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9v6\" (UniqueName: \"kubernetes.io/projected/4d595622-6926-49df-b3f9-682fa21c727d-kube-api-access-nv9v6\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662705 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662781 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e47564-b224-41c0-b438-6c00d3dec913-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662798 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662813 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b6f90f-bf37-48be-aeba-1e6ef73b373b-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662827 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d595622-6926-49df-b3f9-682fa21c727d-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662843 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2bacc7b-11be-442c-b337-815ac913ffe3-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662857 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d595622-6926-49df-b3f9-682fa21c727d-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.662921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662251 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:00:35.662921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.662271 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x" pod "a2bacc7b-11be-442c-b337-815ac913ffe3" (UID: "a2bacc7b-11be-442c-b337-815ac913ffe3"). InnerVolumeSpecName "kube-api-access-h6m6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:00:35.667153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.667132 5710 scope.go:122] "RemoveContainer" containerID="b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2" Nov 01 09:00:35.667532 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:35.667505 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2\": container with ID starting with b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2 not found: ID does not exist" containerID="b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2" Nov 01 09:00:35.667792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.667545 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2"} err="failed to get container status \"b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2\": rpc error: code = NotFound desc = could not find container \"b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2\": container with ID starting with b58ce3296258d76fa3a418db5310e9d355d7f50f2760efd2d92d6383bff443e2 not found: ID does not exist" Nov 01 09:00:35.667792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.667570 5710 scope.go:122] "RemoveContainer" containerID="5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d" Nov 01 09:00:35.691159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.691125 5710 scope.go:122] "RemoveContainer" containerID="5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d" Nov 01 09:00:35.696668 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:35.696633 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d\": container with ID starting with 5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d not found: ID does not exist" containerID="5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d" Nov 01 09:00:35.696787 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.696681 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d"} err="failed to get container status \"5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d\": rpc error: code = NotFound desc = could not find container \"5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d\": container with ID starting with 5ec1bb42da6cf8d95fb20f8ba6053bedb34d53d05f3d34191dcfaa70113fc58d not found: ID does not exist" Nov 01 09:00:35.696787 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.696708 5710 scope.go:122] "RemoveContainer" containerID="08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158" Nov 01 09:00:35.710802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.710576 5710 scope.go:122] "RemoveContainer" containerID="08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158" Nov 01 09:00:35.712071 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:35.711944 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158\": container with ID starting with 08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158 not found: ID does not exist" containerID="08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158" Nov 01 09:00:35.712071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.711983 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158"} err="failed to get container status \"08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158\": rpc error: code = NotFound desc = could not find container \"08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158\": container with ID starting with 08a28b6b657a50d0056fb4e327b8c63e505d4ad5cfc9f31b7b4a13e6c5e89158 not found: ID does not exist" Nov 01 09:00:35.712071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.712008 5710 scope.go:122] "RemoveContainer" containerID="4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81" Nov 01 09:00:35.729100 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.729077 5710 scope.go:122] "RemoveContainer" containerID="4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81" Nov 01 09:00:35.729503 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:35.729470 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81\": container with ID starting with 4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81 not found: ID does not exist" containerID="4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81" Nov 01 09:00:35.729604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.729511 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81"} err="failed to get container status \"4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81\": rpc error: code = NotFound desc = could not find container \"4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81\": container with ID starting with 4df6dafa226ebf2a3843db7618902f95bf8a610802ab92ab6c59496b75e14b81 not found: ID does not exist" Nov 01 09:00:35.729604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.729529 5710 scope.go:122] "RemoveContainer" containerID="332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046" Nov 01 09:00:35.741512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.741486 5710 scope.go:122] "RemoveContainer" containerID="332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046" Nov 01 09:00:35.741986 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:35.741959 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046\": container with ID starting with 332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046 not found: ID does not exist" containerID="332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046" Nov 01 09:00:35.742112 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.741993 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046"} err="failed to get container status \"332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046\": rpc error: code = NotFound desc = could not find container \"332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046\": container with ID starting with 332f0c384bedf878a0ae24394e4ebbc6f93eb440c108453c6bbaaa77326cc046 not found: ID does not exist" Nov 01 09:00:35.763969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.763928 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6m6x\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-kube-api-access-h6m6x\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.763969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.763963 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.763969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.763974 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2bacc7b-11be-442c-b337-815ac913ffe3-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:00:35.865737 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.865625 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj"] Nov 01 09:00:35.872037 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:35.871998 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90022a98_79b2_4156_8ab1_3f7d82dd9490.slice/crio-29be1aac30e42471541b4a3ac4decf483af601c67b90a7f51e3b4075509faa25 WatchSource:0}: Error finding container 29be1aac30e42471541b4a3ac4decf483af601c67b90a7f51e3b4075509faa25: Status 404 returned error can't find the container with id 29be1aac30e42471541b4a3ac4decf483af601c67b90a7f51e3b4075509faa25 Nov 01 09:00:35.875830 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.875784 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:00:35.875944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.875877 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:00:35.875944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.875933 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:00:35.968385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.968352 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc"] Nov 01 09:00:35.974790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:35.974757 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-76rlc"] Nov 01 09:00:36.192042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.191953 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2"] Nov 01 09:00:36.199295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.199244 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-lttc2"] Nov 01 09:00:36.370747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.368160 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b"] Nov 01 09:00:36.375744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.372921 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-p2z6b"] Nov 01 09:00:36.451768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.451668 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8"] Nov 01 09:00:36.457942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.457910 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-4mxq8"] Nov 01 09:00:36.563173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.563129 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd"] Nov 01 09:00:36.568772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.568743 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-8mlfd"] Nov 01 09:00:36.649180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.649131 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" event={"ID":"90022a98-79b2-4156-8ab1-3f7d82dd9490","Type":"ContainerStarted","Data":"c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a"} Nov 01 09:00:36.649401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.649188 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" event={"ID":"90022a98-79b2-4156-8ab1-3f7d82dd9490","Type":"ContainerStarted","Data":"29be1aac30e42471541b4a3ac4decf483af601c67b90a7f51e3b4075509faa25"} Nov 01 09:00:36.655093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.655052 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" event={"ID":"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b","Type":"ContainerStarted","Data":"51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08"} Nov 01 09:00:36.655275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.655110 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:36.764315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.764196 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podStartSLOduration=2.764179935 podStartE2EDuration="2.764179935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:36.761890081 +0000 UTC m=+1659.850226754" watchObservedRunningTime="2026-11-01 09:00:36.764179935 +0000 UTC m=+1659.852516607" Nov 01 09:00:36.910746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:36.903455 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" podStartSLOduration=2.9169876759999998 podStartE2EDuration="3.903429606s" totalImagesPullingTime="986.44193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:36.877284384 +0000 UTC m=+1659.965621058" watchObservedRunningTime="2026-11-01 09:00:36.903429606 +0000 UTC m=+1659.991766279" Nov 01 09:00:37.069611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.069518 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d595622-6926-49df-b3f9-682fa21c727d" path="/var/lib/kubelet/pods/4d595622-6926-49df-b3f9-682fa21c727d/volumes" Nov 01 09:00:37.070311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.070292 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" path="/var/lib/kubelet/pods/82b6f90f-bf37-48be-aeba-1e6ef73b373b/volumes" Nov 01 09:00:37.070929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.070913 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" path="/var/lib/kubelet/pods/a2bacc7b-11be-442c-b337-815ac913ffe3/volumes" Nov 01 09:00:37.072063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.072044 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" path="/var/lib/kubelet/pods/a8e47564-b224-41c0-b438-6c00d3dec913/volumes" Nov 01 09:00:37.072694 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.072677 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" path="/var/lib/kubelet/pods/c0d956d1-cc92-4492-98a0-6e485dc9fd0d/volumes" Nov 01 09:00:37.216216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.216183 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:37.223909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.223878 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:37.663911 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.663879 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:37.664767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:37.664748 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:00:42.669953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:42.669916 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:00:43.602280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602237 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x"] Nov 01 09:00:43.602944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602928 5710 cpu_manager.go:414] "containerMap: removing container" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" Nov 01 09:00:43.602944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602943 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602962 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602968 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602979 5710 cpu_manager.go:414] "containerMap: removing container" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602985 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602992 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.602998 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603009 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603014 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603114 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b6f90f-bf37-48be-aeba-1e6ef73b373b" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603124 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d956d1-cc92-4492-98a0-6e485dc9fd0d" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603130 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d595622-6926-49df-b3f9-682fa21c727d" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603136 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e47564-b224-41c0-b438-6c00d3dec913" containerName="istio-proxy" Nov 01 09:00:43.603141 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.603142 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2bacc7b-11be-442c-b337-815ac913ffe3" containerName="istio-proxy" Nov 01 09:00:43.606959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.606938 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:43.656628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.656593 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-568f63\"" Nov 01 09:00:43.716096 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.716065 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-568f63-dockercfg-xbmrw\"" Nov 01 09:00:43.843784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.718768 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69667\" (UniqueName: \"kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667\") pod \"payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:43.843784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.842444 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:43.861015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.859821 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x"] Nov 01 09:00:43.871957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.871925 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4"] Nov 01 09:00:43.876994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.876966 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4"] Nov 01 09:00:43.877154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.877076 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:43.944101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.944072 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69667\" (UniqueName: \"kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667\") pod \"payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:43.944270 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.944173 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:43.944270 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.944225 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:43.944340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.944324 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jgr\" (UniqueName: \"kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr\") pod \"payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:43.944960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:43.944943 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:44.001877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.001838 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69667\" (UniqueName: \"kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667\") pod \"payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:44.045514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.045475 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:44.045678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.045564 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jgr\" (UniqueName: \"kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr\") pod \"payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:44.046336 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.046314 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:44.117753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.117600 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jgr\" (UniqueName: \"kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr\") pod \"payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:44.192777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.192708 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:44.222564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.222521 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:44.581126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.581040 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4"] Nov 01 09:00:44.587557 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:00:44.587526 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3f4430_ab28_4e09_b8c6_7381eba07e6b.slice/crio-c6c8794be063ae06c25cc980720715b7ebf2b6e1fb8f55009d51628e7e9fc10f WatchSource:0}: Error finding container c6c8794be063ae06c25cc980720715b7ebf2b6e1fb8f55009d51628e7e9fc10f: Status 404 returned error can't find the container with id c6c8794be063ae06c25cc980720715b7ebf2b6e1fb8f55009d51628e7e9fc10f Nov 01 09:00:44.662391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.659809 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x"] Nov 01 09:00:44.858126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.858083 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" event={"ID":"6dd26f75-477a-4ea4-a64c-d059820935c4","Type":"ContainerStarted","Data":"278908e11ed95e8dcf2e6aca577f91e1bdc42112d55ab31708bd1bbf04620e29"} Nov 01 09:00:44.859945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.859914 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" event={"ID":"7e3f4430-ab28-4e09-b8c6-7381eba07e6b","Type":"ContainerStarted","Data":"e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5"} Nov 01 09:00:44.860174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.859960 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" event={"ID":"7e3f4430-ab28-4e09-b8c6-7381eba07e6b","Type":"ContainerStarted","Data":"c6c8794be063ae06c25cc980720715b7ebf2b6e1fb8f55009d51628e7e9fc10f"} Nov 01 09:00:44.860842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.860816 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:44.965130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.965094 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:00:44.970895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:44.970842 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" podStartSLOduration=1.9708281410000001 podStartE2EDuration="1.970828141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:44.968625043 +0000 UTC m=+1668.056961716" watchObservedRunningTime="2026-11-01 09:00:44.970828141 +0000 UTC m=+1668.059164813" Nov 01 09:00:45.491501 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:45.491426 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:00:45.878885 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:45.878839 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" event={"ID":"6dd26f75-477a-4ea4-a64c-d059820935c4","Type":"ContainerStarted","Data":"e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a"} Nov 01 09:00:45.879409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:45.878975 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:46.012746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:46.006138 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" podStartSLOduration=3.006117141 podStartE2EDuration="3.006117141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:00:45.994140984 +0000 UTC m=+1669.082477657" watchObservedRunningTime="2026-11-01 09:00:46.006117141 +0000 UTC m=+1669.094453816" Nov 01 09:00:56.887996 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:56.887967 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:00:56.888438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:56.888034 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:00:58.731920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.731889 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96"] Nov 01 09:00:58.738271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.738248 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.788773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.788702 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.788970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.788839 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.789042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.788994 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgzd\" (UniqueName: \"kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.789157 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.789141 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.792080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.792054 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96"] Nov 01 09:00:58.833930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.833892 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c1f453-serving-cert\"" Nov 01 09:00:58.896597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.896426 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.896597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.896556 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgzd\" (UniqueName: \"kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.896868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.896703 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.896868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.896828 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.901677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.897852 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.901677 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:58.898511 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-serving-cert: secret "maas-api-e2e-api-b-c1f453-serving-cert" not found Nov 01 09:00:58.901677 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:00:58.898624 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls podName:15cb4d9b-f16c-483c-8949-a8fecf0926f8 nodeName:}" failed. No retries permitted until 2026-11-01 09:00:59.398601831 +0000 UTC m=+1682.486938498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls") pod "maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" (UID: "15cb4d9b-f16c-483c-8949-a8fecf0926f8") : secret "maas-api-e2e-api-b-c1f453-serving-cert" not found Nov 01 09:00:58.914476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.910482 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:58.958489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:58.958456 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgzd\" (UniqueName: \"kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:59.409413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:59.409383 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:59.412249 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:59.412227 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\") pod \"maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:00:59.656196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:00:59.656164 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:01:00.037196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:00.037116 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96"] Nov 01 09:01:00.044018 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:01:00.043987 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cb4d9b_f16c_483c_8949_a8fecf0926f8.slice/crio-aa711e68ed4a5165949223daa9f2f4e49b09e355b2b618523a693fa62f863192 WatchSource:0}: Error finding container aa711e68ed4a5165949223daa9f2f4e49b09e355b2b618523a693fa62f863192: Status 404 returned error can't find the container with id aa711e68ed4a5165949223daa9f2f4e49b09e355b2b618523a693fa62f863192 Nov 01 09:01:01.004749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:01.004673 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" event={"ID":"15cb4d9b-f16c-483c-8949-a8fecf0926f8","Type":"ContainerStarted","Data":"3a1fd8ac7307fa28f00f38997d2989169aa6ac9622d09862e4abd805cb057b03"} Nov 01 09:01:01.004963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:01.004765 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" event={"ID":"15cb4d9b-f16c-483c-8949-a8fecf0926f8","Type":"ContainerStarted","Data":"aa711e68ed4a5165949223daa9f2f4e49b09e355b2b618523a693fa62f863192"} Nov 01 09:01:01.004963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:01.004878 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:01:01.117770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:01.117701 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" podStartSLOduration=2.348207388 podStartE2EDuration="3.117685315s" totalImagesPullingTime="769.477927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:01.116423771 +0000 UTC m=+1684.204760483" watchObservedRunningTime="2026-11-01 09:01:01.117685315 +0000 UTC m=+1684.206021987" Nov 01 09:01:03.562794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.562757 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c"] Nov 01 09:01:03.567570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.567538 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.619905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.619870 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-73660a-openshift-default-dockercfg-vztk4\"" Nov 01 09:01:03.671477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.671443 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c"] Nov 01 09:01:03.682472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.682432 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.683878 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.683847 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.684625 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.684603 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.684873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.684844 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.685046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.685030 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.685180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.685165 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.685341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.685314 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.685472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.685429 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.685535 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.685489 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrv9d\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.852946 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853046 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrv9d\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853194 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853261 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853302 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853409 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853478 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853595 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.854385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.853666 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.858538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.857177 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.858538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.857910 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.858538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.858154 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.860950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.860916 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.861681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.861655 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.862964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.862941 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.900203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.900131 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.918183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.918139 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:03.928612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:03.928573 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrv9d\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d\") pod \"e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:04.183704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:04.183621 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:04.656024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:04.655988 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c"] Nov 01 09:01:04.662320 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:01:04.662284 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba49da73_1f72_46fd_affa_a286031ead32.slice/crio-2349b4f990deabf94a9ac9d07ae8d7a1501dba0f934c63475cfa54fa81c948e4 WatchSource:0}: Error finding container 2349b4f990deabf94a9ac9d07ae8d7a1501dba0f934c63475cfa54fa81c948e4: Status 404 returned error can't find the container with id 2349b4f990deabf94a9ac9d07ae8d7a1501dba0f934c63475cfa54fa81c948e4 Nov 01 09:01:04.665084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:04.665027 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:04.665172 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:04.665123 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:04.665172 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:04.665155 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:05.110420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:05.110377 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" event={"ID":"ba49da73-1f72-46fd-affa-a286031ead32","Type":"ContainerStarted","Data":"f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094"} Nov 01 09:01:05.110637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:05.110438 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" event={"ID":"ba49da73-1f72-46fd-affa-a286031ead32","Type":"ContainerStarted","Data":"2349b4f990deabf94a9ac9d07ae8d7a1501dba0f934c63475cfa54fa81c948e4"} Nov 01 09:01:05.183847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:05.183807 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:05.190581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:05.190550 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:05.226440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:05.226381 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podStartSLOduration=2.226362114 podStartE2EDuration="2.226362114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:05.224617079 +0000 UTC m=+1688.312953752" watchObservedRunningTime="2026-11-01 09:01:05.226362114 +0000 UTC m=+1688.314698787" Nov 01 09:01:06.118020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:06.117982 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:06.118950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:06.118929 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:01:07.017638 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:07.017610 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:01:11.573764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.573667 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz"] Nov 01 09:01:11.581282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.581254 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.619614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.619570 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrbx\" (UniqueName: \"kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx\") pod \"payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.619892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.619858 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.631921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.631888 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz"] Nov 01 09:01:11.760773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.752019 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.760773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.752235 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrbx\" (UniqueName: \"kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx\") pod \"payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.761065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.760847 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73660a\"" Nov 01 09:01:11.781788 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.761454 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73660a-dockercfg-hlbwb\"" Nov 01 09:01:11.782034 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.764738 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.783483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.773134 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz"] Nov 01 09:01:11.815743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.796263 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:11.856701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.856657 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz"] Nov 01 09:01:11.879768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.879733 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrbx\" (UniqueName: \"kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx\") pod \"payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.920657 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.920625 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:11.960527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.960482 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrnx\" (UniqueName: \"kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx\") pod \"payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:11.960776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:11.960755 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:12.062685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.062631 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:12.062916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.062887 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrnx\" (UniqueName: \"kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx\") pod \"payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:12.063501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.063477 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:12.121416 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.121330 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrnx\" (UniqueName: \"kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx\") pod \"payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:12.180887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.180846 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:12.304996 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.304960 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz"] Nov 01 09:01:12.311954 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:01:12.311906 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88930cb_d255_4b18_a73f_e4c28560a8d1.slice/crio-4ba3362dc9c5637666de55b2baf36c67ceff723fdc3e14ce0dba1cad46dfbe0c WatchSource:0}: Error finding container 4ba3362dc9c5637666de55b2baf36c67ceff723fdc3e14ce0dba1cad46dfbe0c: Status 404 returned error can't find the container with id 4ba3362dc9c5637666de55b2baf36c67ceff723fdc3e14ce0dba1cad46dfbe0c Nov 01 09:01:12.567970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:12.567929 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz"] Nov 01 09:01:12.574284 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:01:12.574247 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode637f02a_2a9b_4272_97ed_2af0b6b72080.slice/crio-eacfc597513c839faac4fae4bd55f0d22e1017d6fbdd25edddcabcc39c2f735d WatchSource:0}: Error finding container eacfc597513c839faac4fae4bd55f0d22e1017d6fbdd25edddcabcc39c2f735d: Status 404 returned error can't find the container with id eacfc597513c839faac4fae4bd55f0d22e1017d6fbdd25edddcabcc39c2f735d Nov 01 09:01:13.195183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.195141 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" event={"ID":"e637f02a-2a9b-4272-97ed-2af0b6b72080","Type":"ContainerStarted","Data":"8c7f625224749ad3adaa8e01c3b2259764b2280888510b6ce1b0d896451e2cdf"} Nov 01 09:01:13.195364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.195200 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" event={"ID":"e637f02a-2a9b-4272-97ed-2af0b6b72080","Type":"ContainerStarted","Data":"eacfc597513c839faac4fae4bd55f0d22e1017d6fbdd25edddcabcc39c2f735d"} Nov 01 09:01:13.195364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.195234 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:13.196827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.196806 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" event={"ID":"c88930cb-d255-4b18-a73f-e4c28560a8d1","Type":"ContainerStarted","Data":"fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449"} Nov 01 09:01:13.196943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.196848 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" event={"ID":"c88930cb-d255-4b18-a73f-e4c28560a8d1","Type":"ContainerStarted","Data":"4ba3362dc9c5637666de55b2baf36c67ceff723fdc3e14ce0dba1cad46dfbe0c"} Nov 01 09:01:13.196943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.196869 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:13.304629 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.304574 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" podStartSLOduration=2.3045596870000002 podStartE2EDuration="2.304559687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:13.304178505 +0000 UTC m=+1696.392515179" watchObservedRunningTime="2026-11-01 09:01:13.304559687 +0000 UTC m=+1696.392896367" Nov 01 09:01:13.447196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.447068 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" podStartSLOduration=2.447047676 podStartE2EDuration="2.447047676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:13.436490281 +0000 UTC m=+1696.524827025" watchObservedRunningTime="2026-11-01 09:01:13.447047676 +0000 UTC m=+1696.535384349" Nov 01 09:01:13.462242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.462200 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7"] Nov 01 09:01:13.462762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:13.462702 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" podUID="b087af6c-ac01-4251-8210-913b3e18122e" containerName="maas-api" containerID="cri-o://93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4" gracePeriod=30 Nov 01 09:01:14.048700 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.046939 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 09:01:14.137453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.137414 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs\") pod \"b087af6c-ac01-4251-8210-913b3e18122e\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " Nov 01 09:01:14.137669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.137490 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\") pod \"b087af6c-ac01-4251-8210-913b3e18122e\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " Nov 01 09:01:14.137669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.137580 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd\" (UniqueName: \"kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd\") pod \"b087af6c-ac01-4251-8210-913b3e18122e\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " Nov 01 09:01:14.137669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.137624 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca\") pod \"b087af6c-ac01-4251-8210-913b3e18122e\" (UID: \"b087af6c-ac01-4251-8210-913b3e18122e\") " Nov 01 09:01:14.138063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.138029 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca" pod "b087af6c-ac01-4251-8210-913b3e18122e" (UID: "b087af6c-ac01-4251-8210-913b3e18122e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:01:14.140217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.140180 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd" pod "b087af6c-ac01-4251-8210-913b3e18122e" (UID: "b087af6c-ac01-4251-8210-913b3e18122e"). InnerVolumeSpecName "kube-api-access-xr8rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:01:14.140415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.140396 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls" pod "b087af6c-ac01-4251-8210-913b3e18122e" (UID: "b087af6c-ac01-4251-8210-913b3e18122e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:01:14.140505 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.140477 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs" pod "b087af6c-ac01-4251-8210-913b3e18122e" (UID: "b087af6c-ac01-4251-8210-913b3e18122e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:01:14.206966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.206928 5710 generic.go:356] "Generic (PLEG): container finished" podID="b087af6c-ac01-4251-8210-913b3e18122e" containerID="93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4" exitCode=0 Nov 01 09:01:14.207169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.207008 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" Nov 01 09:01:14.207169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.207030 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" event={"ID":"b087af6c-ac01-4251-8210-913b3e18122e","Type":"ContainerDied","Data":"93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4"} Nov 01 09:01:14.207169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.207094 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7" event={"ID":"b087af6c-ac01-4251-8210-913b3e18122e","Type":"ContainerDied","Data":"1cb1f98a4d0e3954984acb79ce3985e4b7dd956a143891110cb9479a93efd549"} Nov 01 09:01:14.207169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.207125 5710 scope.go:122] "RemoveContainer" containerID="93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4" Nov 01 09:01:14.223598 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.223571 5710 scope.go:122] "RemoveContainer" containerID="93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4" Nov 01 09:01:14.224030 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:14.224001 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4\": container with ID starting with 93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4 not found: ID does not exist" containerID="93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4" Nov 01 09:01:14.224151 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.224045 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4"} err="failed to get container status \"93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4\": rpc error: code = NotFound desc = could not find container \"93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4\": container with ID starting with 93c2cdc30f309bdb5822bf168faf04eedbe7a9694c87b6e6f92d206a018201f4 not found: ID does not exist" Nov 01 09:01:14.239318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.239282 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b087af6c-ac01-4251-8210-913b3e18122e-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:01:14.239318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.239318 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:01:14.239318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.239335 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b087af6c-ac01-4251-8210-913b3e18122e-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:01:14.239605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.239349 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8rd\" (UniqueName: \"kubernetes.io/projected/b087af6c-ac01-4251-8210-913b3e18122e-kube-api-access-xr8rd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:01:14.364611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.364575 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7"] Nov 01 09:01:14.370780 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:14.370746 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d0ba8f-78595fc7b5-wnth7"] Nov 01 09:01:15.061441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:15.061401 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b087af6c-ac01-4251-8210-913b3e18122e" path="/var/lib/kubelet/pods/b087af6c-ac01-4251-8210-913b3e18122e/volumes" Nov 01 09:01:15.538848 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:15.538807 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:01:24.211756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.211728 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:01:24.217233 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.211808 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:01:24.985561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.985518 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp"] Nov 01 09:01:24.986406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.986381 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b087af6c-ac01-4251-8210-913b3e18122e" containerName="maas-api" Nov 01 09:01:24.986535 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.986408 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b087af6c-ac01-4251-8210-913b3e18122e" containerName="maas-api" Nov 01 09:01:24.986651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.986632 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b087af6c-ac01-4251-8210-913b3e18122e" containerName="maas-api" Nov 01 09:01:24.990505 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:24.990481 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035028 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035104 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035143 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035231 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035279 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035362 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzz9\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035406 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035436 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.035514 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.054756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.053812 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-991c28-openshift-default-dockercfg-tvzzv\"" Nov 01 09:01:25.184641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184605 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzz9\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.184912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184661 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.184912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184690 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.184912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184777 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.184912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184892 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.185136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184937 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.185136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.184976 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.185136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.185054 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.185136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.185104 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.191213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.191181 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.339793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.322038 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.339793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.322657 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.339793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.323284 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.339793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.324911 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.339793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.325067 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.339793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.326364 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.340414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.340133 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp"] Nov 01 09:01:25.348084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.348058 5710 generic.go:356] "Generic (PLEG): container finished" podID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerID="ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd" exitCode=0 Nov 01 09:01:25.348249 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.348132 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" event={"ID":"9ac479b6-2bb8-410e-aca6-d3791e9f8744","Type":"ContainerDied","Data":"ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd"} Nov 01 09:01:25.348567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.348549 5710 scope.go:122] "RemoveContainer" containerID="ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd" Nov 01 09:01:25.350243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.350215 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:25.350331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.350284 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:25.350331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.350317 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:25.388709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.387696 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.389008 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.388986 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzz9\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9\") pod \"e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:25.634745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:25.631975 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:26.080093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.080002 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp"] Nov 01 09:01:26.112257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.112220 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:26.112396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.112296 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:26.112396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.112326 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:26.357016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.356975 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" event={"ID":"bac2d1a7-18ca-4294-b82c-4607fcad47d0","Type":"ContainerStarted","Data":"1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4"} Nov 01 09:01:26.357459 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.357044 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" event={"ID":"bac2d1a7-18ca-4294-b82c-4607fcad47d0","Type":"ContainerStarted","Data":"e205b4ea3246a29a4c3b16d76e5eabb2a59111fc84f9661a755cba2d40e91b88"} Nov 01 09:01:26.359577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.359552 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" event={"ID":"9ac479b6-2bb8-410e-aca6-d3791e9f8744","Type":"ContainerStarted","Data":"7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80"} Nov 01 09:01:26.473486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.473432 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podStartSLOduration=2.473415818 podStartE2EDuration="2.473415818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:26.471110768 +0000 UTC m=+1709.559447441" watchObservedRunningTime="2026-11-01 09:01:26.473415818 +0000 UTC m=+1709.561752491" Nov 01 09:01:26.641424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:26.641282 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:27.038702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:27.038622 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:01:27.038929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:27.038881 5710 patch_prober.go:29] interesting pod/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 09:01:27.038972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:27.038927 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 09:01:27.638208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:27.638174 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:28.039863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:28.039769 5710 patch_prober.go:29] interesting pod/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 09:01:28.039863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:28.039829 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 09:01:28.374025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:28.373993 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:28.375186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:28.375165 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:01:29.039361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:29.039317 5710 patch_prober.go:29] interesting pod/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 09:01:29.039928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:29.039368 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 09:01:30.039897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.039862 5710 patch_prober.go:29] interesting pod/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 09:01:30.040316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.039929 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 09:01:30.248183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.248144 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv"] Nov 01 09:01:30.253796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.253772 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.312455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.312359 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv"] Nov 01 09:01:30.336592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.336558 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.336592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.336597 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.337056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.336983 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.337056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.337045 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxq94\" (UniqueName: \"kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.354257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.354232 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-568f63-serving-cert\"" Nov 01 09:01:30.440347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.440309 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.440767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.440739 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxq94\" (UniqueName: \"kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.440892 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:30.440660 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-serving-cert: secret "maas-api-e2e-api-b-568f63-serving-cert" not found Nov 01 09:01:30.440942 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:30.440927 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls podName:d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059 nodeName:}" failed. No retries permitted until 2026-11-01 09:01:30.940903576 +0000 UTC m=+1714.029240228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls") pod "maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" (UID: "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059") : secret "maas-api-e2e-api-b-568f63-serving-cert" not found Nov 01 09:01:30.441310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.441288 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.441359 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.441332 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.442605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.442580 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.444858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.444320 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.508698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.508662 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxq94\" (UniqueName: \"kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.960172 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.960134 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:30.962832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:30.962806 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\") pod \"maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:31.039344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:31.039294 5710 patch_prober.go:29] interesting pod/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 09:01:31.039567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:31.039357 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 09:01:31.169796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:31.169767 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:31.577281 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:31.577196 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv"] Nov 01 09:01:31.607172 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:01:31.607134 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45cbfa7_2fff_4563_ac5c_8a7e5d6e4059.slice/crio-4018ab80ee7952961410e8d0138d14010639b0a43039ef903df6f839b71f0e3e WatchSource:0}: Error finding container 4018ab80ee7952961410e8d0138d14010639b0a43039ef903df6f839b71f0e3e: Status 404 returned error can't find the container with id 4018ab80ee7952961410e8d0138d14010639b0a43039ef903df6f839b71f0e3e Nov 01 09:01:32.043811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:32.043778 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:01:32.044227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:32.044205 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:01:32.045065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:32.045047 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:01:32.432867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:32.432830 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" event={"ID":"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059","Type":"ContainerStarted","Data":"4018ab80ee7952961410e8d0138d14010639b0a43039ef903df6f839b71f0e3e"} Nov 01 09:01:33.443135 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:33.443094 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" event={"ID":"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059","Type":"ContainerStarted","Data":"cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b"} Nov 01 09:01:33.443623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:33.443192 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:33.553917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:33.553834 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" podStartSLOduration=2.699772583 podStartE2EDuration="3.553813592s" totalImagesPullingTime="854.041009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:33.553250002 +0000 UTC m=+1716.641586675" watchObservedRunningTime="2026-11-01 09:01:33.553813592 +0000 UTC m=+1716.642150265" Nov 01 09:01:36.481800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.481702 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7409-9d96f1"] Nov 01 09:01:36.487558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.487534 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:36.537184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.537149 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7409-9d96f1"] Nov 01 09:01:36.578063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.578027 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-44wvn\"" Nov 01 09:01:36.658534 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.658497 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlv9\" (UniqueName: \"kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9\") pod \"test-curl-7409-9d96f1\" (UID: \"26b4b549-5dc2-4efc-a223-86521fe4ce37\") " pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:36.760244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.760137 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlv9\" (UniqueName: \"kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9\") pod \"test-curl-7409-9d96f1\" (UID: \"26b4b549-5dc2-4efc-a223-86521fe4ce37\") " pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:36.817683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:36.817650 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlv9\" (UniqueName: \"kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9\") pod \"test-curl-7409-9d96f1\" (UID: \"26b4b549-5dc2-4efc-a223-86521fe4ce37\") " pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:37.100668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:37.100634 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:37.489923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:37.489835 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7409-9d96f1"] Nov 01 09:01:37.496304 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:01:37.496269 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26b4b549_5dc2_4efc_a223_86521fe4ce37.slice/crio-f8dc8e0ce8560053d09155ff88cb808276908421dbfa74185898a4ded685e4af WatchSource:0}: Error finding container f8dc8e0ce8560053d09155ff88cb808276908421dbfa74185898a4ded685e4af: Status 404 returned error can't find the container with id f8dc8e0ce8560053d09155ff88cb808276908421dbfa74185898a4ded685e4af Nov 01 09:01:38.630399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:38.630359 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7409-9d96f1" event={"ID":"26b4b549-5dc2-4efc-a223-86521fe4ce37","Type":"ContainerStarted","Data":"f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227"} Nov 01 09:01:38.630942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:38.630411 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7409-9d96f1" event={"ID":"26b4b549-5dc2-4efc-a223-86521fe4ce37","Type":"ContainerStarted","Data":"f8dc8e0ce8560053d09155ff88cb808276908421dbfa74185898a4ded685e4af"} Nov 01 09:01:38.740506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:38.740435 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7409-9d96f1" podStartSLOduration=2.740420098 podStartE2EDuration="2.740420098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:01:38.737105991 +0000 UTC m=+1721.825442667" watchObservedRunningTime="2026-11-01 09:01:38.740420098 +0000 UTC m=+1721.828756769" Nov 01 09:01:39.455684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:39.455652 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:01:40.243070 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:40.243031 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7409-9d96f1"] Nov 01 09:01:40.247741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:40.247694 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7409-9d96f1"] Nov 01 09:01:40.646602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:40.646556 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7409-9d96f1" podUID="26b4b549-5dc2-4efc-a223-86521fe4ce37" containerName="test-curl-7409-9d96f1" containerID="cri-o://f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227" gracePeriod=2 Nov 01 09:01:40.694915 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:40.694862 5710 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7409-9d96f1\" is forbidden: User \"system:node:ip-10-0-143-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-16.us-west-2.compute.internal' and this object" podUID="26b4b549-5dc2-4efc-a223-86521fe4ce37" pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:42.888250 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:42.888223 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:42.949655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:42.949563 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9\" (UniqueName: \"kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9\") pod \"26b4b549-5dc2-4efc-a223-86521fe4ce37\" (UID: \"26b4b549-5dc2-4efc-a223-86521fe4ce37\") " Nov 01 09:01:42.951933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:42.951910 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9" pod "26b4b549-5dc2-4efc-a223-86521fe4ce37" (UID: "26b4b549-5dc2-4efc-a223-86521fe4ce37"). InnerVolumeSpecName "kube-api-access-lxlv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:01:43.052149 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.052105 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlv9\" (UniqueName: \"kubernetes.io/projected/26b4b549-5dc2-4efc-a223-86521fe4ce37-kube-api-access-lxlv9\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:01:43.060095 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.060068 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b4b549-5dc2-4efc-a223-86521fe4ce37" path="/var/lib/kubelet/pods/26b4b549-5dc2-4efc-a223-86521fe4ce37/volumes" Nov 01 09:01:43.673448 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.673403 5710 generic.go:356] "Generic (PLEG): container finished" podID="26b4b549-5dc2-4efc-a223-86521fe4ce37" containerID="f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227" exitCode=137 Nov 01 09:01:43.673664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.673563 5710 scope.go:122] "RemoveContainer" containerID="f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227" Nov 01 09:01:43.674051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.674026 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-9d96f1" Nov 01 09:01:43.684852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.684831 5710 scope.go:122] "RemoveContainer" containerID="f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227" Nov 01 09:01:43.685227 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:43.685204 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227\": container with ID starting with f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227 not found: ID does not exist" containerID="f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227" Nov 01 09:01:43.685334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:43.685234 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227"} err="failed to get container status \"f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227\": rpc error: code = NotFound desc = could not find container \"f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227\": container with ID starting with f10afffd341527961140f7c26d34b979f180391f8711c3e3279d889a0b57e227 not found: ID does not exist" Nov 01 09:01:44.659926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:44.659894 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:01:45.574980 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:01:45.574936 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:01:53.705895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:53.705850 5710 patch_prober.go:29] interesting pod/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:01:53.706350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:53.705928 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:01:55.781190 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:55.781157 5710 generic.go:356] "Generic (PLEG): container finished" podID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerID="c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a" exitCode=0 Nov 01 09:01:55.781627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:55.781233 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" event={"ID":"90022a98-79b2-4156-8ab1-3f7d82dd9490","Type":"ContainerDied","Data":"c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a"} Nov 01 09:01:55.781765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:55.781745 5710 scope.go:122] "RemoveContainer" containerID="c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a" Nov 01 09:01:55.783587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:55.783559 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:55.783652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:55.783631 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:55.783689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:55.783664 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:01:56.812457 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:56.812410 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" event={"ID":"90022a98-79b2-4156-8ab1-3f7d82dd9490","Type":"ContainerStarted","Data":"eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af"} Nov 01 09:01:57.215956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:57.215924 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:01:57.216193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:57.216169 5710 patch_prober.go:29] interesting pod/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 09:01:57.216274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:57.216222 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 09:01:58.216379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:58.216342 5710 patch_prober.go:29] interesting pod/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 09:01:58.216880 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:58.216402 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 09:01:59.216135 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:59.216093 5710 patch_prober.go:29] interesting pod/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 09:01:59.216326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:01:59.216163 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 09:02:00.216309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:00.216264 5710 patch_prober.go:29] interesting pod/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 09:02:00.216777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:00.216347 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 09:02:01.217773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:01.217691 5710 patch_prober.go:29] interesting pod/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 09:02:01.218295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:01.217802 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 09:02:02.220838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:02.220806 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:02:02.221376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:02.221104 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:02:02.222067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:02.222040 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:02:06.011786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.011702 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk"] Nov 01 09:02:06.012524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.012504 5710 cpu_manager.go:414] "containerMap: removing container" podUID="26b4b549-5dc2-4efc-a223-86521fe4ce37" containerName="test-curl-7409-9d96f1" Nov 01 09:02:06.012579 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.012526 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b4b549-5dc2-4efc-a223-86521fe4ce37" containerName="test-curl-7409-9d96f1" Nov 01 09:02:06.012703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.012692 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b4b549-5dc2-4efc-a223-86521fe4ce37" containerName="test-curl-7409-9d96f1" Nov 01 09:02:06.018065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.018035 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.067300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.067270 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-73660a-serving-cert\"" Nov 01 09:02:06.071449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.071414 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk"] Nov 01 09:02:06.185046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.185012 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.185233 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.185201 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.185284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.185267 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.185540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.185523 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtjs\" (UniqueName: \"kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.290817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.290668 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.290817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.290791 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.291090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.290990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtjs\" (UniqueName: \"kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.291090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.291024 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.295920 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:06.291241 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-serving-cert: secret "maas-api-e2e-ipp-a-73660a-serving-cert" not found Nov 01 09:02:06.295920 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:06.291332 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls podName:61c26a0c-731c-4593-b1ef-11192fae18fe nodeName:}" failed. No retries permitted until 2026-11-01 09:02:06.791309061 +0000 UTC m=+1749.879645731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls") pod "maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" (UID: "61c26a0c-731c-4593-b1ef-11192fae18fe") : secret "maas-api-e2e-ipp-a-73660a-serving-cert" not found Nov 01 09:02:06.295920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.292416 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.320209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.320170 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.351180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.351150 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtjs\" (UniqueName: \"kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.800115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.800082 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.802889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.802869 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:06.933352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:06.933312 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:07.314594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:07.314561 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk"] Nov 01 09:02:07.321263 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:02:07.321235 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c26a0c_731c_4593_b1ef_11192fae18fe.slice/crio-5d8772e9c091020aa73e3f9c177afc0c9ca2e315515ddc804f9cf9af952b7460 WatchSource:0}: Error finding container 5d8772e9c091020aa73e3f9c177afc0c9ca2e315515ddc804f9cf9af952b7460: Status 404 returned error can't find the container with id 5d8772e9c091020aa73e3f9c177afc0c9ca2e315515ddc804f9cf9af952b7460 Nov 01 09:02:07.889793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:07.889753 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" event={"ID":"61c26a0c-731c-4593-b1ef-11192fae18fe","Type":"ContainerStarted","Data":"5d8772e9c091020aa73e3f9c177afc0c9ca2e315515ddc804f9cf9af952b7460"} Nov 01 09:02:08.900049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:08.900014 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" event={"ID":"61c26a0c-731c-4593-b1ef-11192fae18fe","Type":"ContainerStarted","Data":"e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed"} Nov 01 09:02:08.900539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:08.900186 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:09.012872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:09.012808 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" podStartSLOduration=3.048147035 podStartE2EDuration="4.012793434s" totalImagesPullingTime="964.646399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:02:09.011921762 +0000 UTC m=+1752.100258446" watchObservedRunningTime="2026-11-01 09:02:09.012793434 +0000 UTC m=+1752.101130146" Nov 01 09:02:12.968193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:12.968154 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w"] Nov 01 09:02:14.913180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:14.913147 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:02:15.477626 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:15.477584 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:02:17.969338 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:17.969282 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" podUID="3e11f3da-7f16-429c-a72b-0978781d6cdc" containerName="payload-pre-processing" containerID="cri-o://4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9" gracePeriod=25 Nov 01 09:02:18.249545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.249517 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 09:02:18.361837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.361789 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p\" (UniqueName: \"kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p\") pod \"3e11f3da-7f16-429c-a72b-0978781d6cdc\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " Nov 01 09:02:18.362818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.361895 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume\") pod \"3e11f3da-7f16-429c-a72b-0978781d6cdc\" (UID: \"3e11f3da-7f16-429c-a72b-0978781d6cdc\") " Nov 01 09:02:18.363024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.362992 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume" pod "3e11f3da-7f16-429c-a72b-0978781d6cdc" (UID: "3e11f3da-7f16-429c-a72b-0978781d6cdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:02:18.365851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.365821 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p" pod "3e11f3da-7f16-429c-a72b-0978781d6cdc" (UID: "3e11f3da-7f16-429c-a72b-0978781d6cdc"). InnerVolumeSpecName "kube-api-access-smz5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:02:18.464301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.464261 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e11f3da-7f16-429c-a72b-0978781d6cdc-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:18.464301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:18.464300 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smz5p\" (UniqueName: \"kubernetes.io/projected/3e11f3da-7f16-429c-a72b-0978781d6cdc-kube-api-access-smz5p\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:19.003575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.003539 5710 generic.go:356] "Generic (PLEG): container finished" podID="3e11f3da-7f16-429c-a72b-0978781d6cdc" containerID="4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9" exitCode=0 Nov 01 09:02:19.004020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.003603 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" Nov 01 09:02:19.004020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.003618 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" event={"ID":"3e11f3da-7f16-429c-a72b-0978781d6cdc","Type":"ContainerDied","Data":"4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9"} Nov 01 09:02:19.004020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.003696 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w" event={"ID":"3e11f3da-7f16-429c-a72b-0978781d6cdc","Type":"ContainerDied","Data":"ae8e2ba96553b4bc4a8afea9f4b07e2b6e6703838ad292702dcad939b71ac9e9"} Nov 01 09:02:19.004020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.003736 5710 scope.go:122] "RemoveContainer" containerID="4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9" Nov 01 09:02:19.014896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.014874 5710 scope.go:122] "RemoveContainer" containerID="4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9" Nov 01 09:02:19.015194 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:19.015175 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9\": container with ID starting with 4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9 not found: ID does not exist" containerID="4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9" Nov 01 09:02:19.015296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.015205 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9"} err="failed to get container status \"4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9\": rpc error: code = NotFound desc = could not find container \"4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9\": container with ID starting with 4eecb9c449c26766fe8f71188fbf54b38b3038b936e67c57d3955c3dcf3db2b9 not found: ID does not exist" Nov 01 09:02:19.158732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.158692 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w"] Nov 01 09:02:19.205203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:19.205159 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03832c-5f584b5587-j6w7w"] Nov 01 09:02:21.060417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:21.060382 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e11f3da-7f16-429c-a72b-0978781d6cdc" path="/var/lib/kubelet/pods/3e11f3da-7f16-429c-a72b-0978781d6cdc/volumes" Nov 01 09:02:23.422099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:23.422058 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg"] Nov 01 09:02:23.523258 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:23.523163 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx"] Nov 01 09:02:25.509327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:25.509286 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr"] Nov 01 09:02:25.509776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:25.509582 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" podUID="a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" containerName="maas-api" containerID="cri-o://51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08" gracePeriod=30 Nov 01 09:02:25.873665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:25.873641 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:02:25.998095 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:25.998059 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg\" (UniqueName: \"kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg\") pod \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " Nov 01 09:02:25.998711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:25.998680 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs\") pod \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " Nov 01 09:02:26.014158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.014130 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca\") pod \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " Nov 01 09:02:26.014477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.014456 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\") pod \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\" (UID: \"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b\") " Nov 01 09:02:26.017539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.013398 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs" pod "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" (UID: "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:02:26.030747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.029371 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca" pod "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" (UID: "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:02:26.042812 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.042772 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls" pod "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" (UID: "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:02:26.043024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.042883 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg" pod "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" (UID: "a1471ad3-a93f-4a83-bd17-5e3008d2cf2b"). InnerVolumeSpecName "kube-api-access-7mmvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:02:26.132795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.122137 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:26.132795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.122176 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:26.132795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.122191 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:26.132795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.122205 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmvg\" (UniqueName: \"kubernetes.io/projected/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b-kube-api-access-7mmvg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:26.156729 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.156675 5710 generic.go:356] "Generic (PLEG): container finished" podID="a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" containerID="51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08" exitCode=0 Nov 01 09:02:26.157010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.156984 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" event={"ID":"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b","Type":"ContainerDied","Data":"51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08"} Nov 01 09:02:26.157460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.157424 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" event={"ID":"a1471ad3-a93f-4a83-bd17-5e3008d2cf2b","Type":"ContainerDied","Data":"fb6a37047b472d5d8e301ca28bc8e16d0ab8fffe9cec6ddcd1114fd2815e97f3"} Nov 01 09:02:26.157637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.157613 5710 scope.go:122] "RemoveContainer" containerID="51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08" Nov 01 09:02:26.158024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.157998 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr" Nov 01 09:02:26.188930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.188906 5710 scope.go:122] "RemoveContainer" containerID="51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08" Nov 01 09:02:26.189318 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:26.189294 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08\": container with ID starting with 51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08 not found: ID does not exist" containerID="51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08" Nov 01 09:02:26.189439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.189328 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08"} err="failed to get container status \"51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08\": rpc error: code = NotFound desc = could not find container \"51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08\": container with ID starting with 51a2d76641fca70ce26fbfd4fd9c4c528d9a95c20a64f5e12aab06dc7e324b08 not found: ID does not exist" Nov 01 09:02:26.319017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.318973 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr"] Nov 01 09:02:26.325010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:26.324978 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d31acd45-698d86ddbf-56tsr"] Nov 01 09:02:27.030959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.030921 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:02:27.062297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.062257 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" path="/var/lib/kubelet/pods/a1471ad3-a93f-4a83-bd17-5e3008d2cf2b/volumes" Nov 01 09:02:27.203663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.203626 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf"] Nov 01 09:02:27.204478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.204455 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3e11f3da-7f16-429c-a72b-0978781d6cdc" containerName="payload-pre-processing" Nov 01 09:02:27.204478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.204473 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e11f3da-7f16-429c-a72b-0978781d6cdc" containerName="payload-pre-processing" Nov 01 09:02:27.204666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.204496 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" containerName="maas-api" Nov 01 09:02:27.204666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.204501 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" containerName="maas-api" Nov 01 09:02:27.204666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.204588 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e11f3da-7f16-429c-a72b-0978781d6cdc" containerName="payload-pre-processing" Nov 01 09:02:27.204666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.204600 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1471ad3-a93f-4a83-bd17-5e3008d2cf2b" containerName="maas-api" Nov 01 09:02:27.209528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.209508 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.258880 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.258834 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-991c28-serving-cert\"" Nov 01 09:02:27.263358 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.263322 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf"] Nov 01 09:02:27.355360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.354608 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.355762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.355701 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2md5g\" (UniqueName: \"kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.355905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.355876 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.355982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.355911 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.457125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.457085 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.457339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.457183 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2md5g\" (UniqueName: \"kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.457339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.457277 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.457339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.457299 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.458106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.458028 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.459886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.459863 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.460075 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.460049 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.515166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.515135 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2md5g\" (UniqueName: \"kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g\") pod \"maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.535711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.535669 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:27.948590 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:27.946995 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf"] Nov 01 09:02:28.265808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.265006 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" event={"ID":"b201ed59-c659-494f-902c-848b18ee8c68","Type":"ContainerStarted","Data":"7102e45756436a865c6cb258aa41cb82bebc731b4b6b7fa46ee83f616785a8f0"} Nov 01 09:02:28.423964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.423875 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" podUID="58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" containerName="payload-processing" containerID="cri-o://c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e" gracePeriod=25 Nov 01 09:02:28.524489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.524384 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" podUID="ddef433d-68ce-4ef7-a1f2-772a7db251e1" containerName="payload-pre-processing" containerID="cri-o://845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e" gracePeriod=25 Nov 01 09:02:28.739494 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.739184 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 09:02:28.828393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.828352 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 09:02:28.901506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.901463 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw\" (UniqueName: \"kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw\") pod \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " Nov 01 09:02:28.901712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.901550 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume\") pod \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\" (UID: \"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6\") " Nov 01 09:02:28.902341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.902298 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume" pod "58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" (UID: "58585a6d-4a10-49dd-ab74-8d0d3cac2fb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:02:28.904593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:28.904566 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw" pod "58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" (UID: "58585a6d-4a10-49dd-ab74-8d0d3cac2fb6"). InnerVolumeSpecName "kube-api-access-h5jdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:02:29.009827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.003763 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume\") pod \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " Nov 01 09:02:29.009827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.003890 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t\" (UniqueName: \"kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t\") pod \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\" (UID: \"ddef433d-68ce-4ef7-a1f2-772a7db251e1\") " Nov 01 09:02:29.009827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.004511 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jdw\" (UniqueName: \"kubernetes.io/projected/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-kube-api-access-h5jdw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:29.009827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.004533 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:29.010672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.010593 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume" pod "ddef433d-68ce-4ef7-a1f2-772a7db251e1" (UID: "ddef433d-68ce-4ef7-a1f2-772a7db251e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:02:29.012454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.012304 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t" pod "ddef433d-68ce-4ef7-a1f2-772a7db251e1" (UID: "ddef433d-68ce-4ef7-a1f2-772a7db251e1"). InnerVolumeSpecName "kube-api-access-fwn6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:02:29.106053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.106001 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddef433d-68ce-4ef7-a1f2-772a7db251e1-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:29.106053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.106040 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwn6t\" (UniqueName: \"kubernetes.io/projected/ddef433d-68ce-4ef7-a1f2-772a7db251e1-kube-api-access-fwn6t\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:29.273209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.273167 5710 generic.go:356] "Generic (PLEG): container finished" podID="58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" containerID="c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e" exitCode=0 Nov 01 09:02:29.273682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.273214 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" Nov 01 09:02:29.273682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.273255 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" event={"ID":"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6","Type":"ContainerDied","Data":"c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e"} Nov 01 09:02:29.273682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.273307 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg" event={"ID":"58585a6d-4a10-49dd-ab74-8d0d3cac2fb6","Type":"ContainerDied","Data":"bc063be7f0b6708da45cb64e5e3ac909eed4aef2941c25832e8108e2f650ba18"} Nov 01 09:02:29.273682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.273338 5710 scope.go:122] "RemoveContainer" containerID="c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e" Nov 01 09:02:29.274842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.274817 5710 generic.go:356] "Generic (PLEG): container finished" podID="ddef433d-68ce-4ef7-a1f2-772a7db251e1" containerID="845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e" exitCode=0 Nov 01 09:02:29.274950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.274870 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" Nov 01 09:02:29.274950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.274906 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" event={"ID":"ddef433d-68ce-4ef7-a1f2-772a7db251e1","Type":"ContainerDied","Data":"845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e"} Nov 01 09:02:29.275117 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.275100 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx" event={"ID":"ddef433d-68ce-4ef7-a1f2-772a7db251e1","Type":"ContainerDied","Data":"a1b7375a01416c3e4f9633afe893db2cd37517016a86aa67c9d97c23cb88ef55"} Nov 01 09:02:29.277797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.277769 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" event={"ID":"b201ed59-c659-494f-902c-848b18ee8c68","Type":"ContainerStarted","Data":"934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069"} Nov 01 09:02:29.277965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.277945 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:29.308796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.308695 5710 scope.go:122] "RemoveContainer" containerID="c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e" Nov 01 09:02:29.309167 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:29.309143 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e\": container with ID starting with c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e not found: ID does not exist" containerID="c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e" Nov 01 09:02:29.309224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.309177 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e"} err="failed to get container status \"c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e\": rpc error: code = NotFound desc = could not find container \"c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e\": container with ID starting with c81e156d46288667723ad1cb6fe7968f594f3d6f8fc77ac9817c7d049307c88e not found: ID does not exist" Nov 01 09:02:29.309224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.309196 5710 scope.go:122] "RemoveContainer" containerID="845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e" Nov 01 09:02:29.319064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.319041 5710 scope.go:122] "RemoveContainer" containerID="845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e" Nov 01 09:02:29.319404 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:29.319381 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e\": container with ID starting with 845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e not found: ID does not exist" containerID="845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e" Nov 01 09:02:29.319516 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.319409 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e"} err="failed to get container status \"845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e\": rpc error: code = NotFound desc = could not find container \"845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e\": container with ID starting with 845c0b000d3f8cbdf1d2fe038bf268d1b1dab7e344b34f7479c78c3b1bc4c13e not found: ID does not exist" Nov 01 09:02:29.430252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.430216 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg"] Nov 01 09:02:29.476583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.476547 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d31acd45-59658f777c-sswfg"] Nov 01 09:02:29.567742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.564539 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx"] Nov 01 09:02:29.572684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.571417 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d31acd45-5979cf855f-ndrxx"] Nov 01 09:02:29.653480 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:29.653420 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" podStartSLOduration=1.825396262 podStartE2EDuration="2.653404034s" totalImagesPullingTime="828.007772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:02:29.650604488 +0000 UTC m=+1772.738941163" watchObservedRunningTime="2026-11-01 09:02:29.653404034 +0000 UTC m=+1772.741740705" Nov 01 09:02:31.060291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:31.060259 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" path="/var/lib/kubelet/pods/58585a6d-4a10-49dd-ab74-8d0d3cac2fb6/volumes" Nov 01 09:02:31.060711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:31.060702 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddef433d-68ce-4ef7-a1f2-772a7db251e1" path="/var/lib/kubelet/pods/ddef433d-68ce-4ef7-a1f2-772a7db251e1/volumes" Nov 01 09:02:35.315824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:35.315792 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:02:37.159943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:37.159901 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:02:37.160385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:37.159987 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:02:39.386327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:39.386286 5710 generic.go:356] "Generic (PLEG): container finished" podID="ba49da73-1f72-46fd-affa-a286031ead32" containerID="f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094" exitCode=0 Nov 01 09:02:39.386872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:39.386374 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" event={"ID":"ba49da73-1f72-46fd-affa-a286031ead32","Type":"ContainerDied","Data":"f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094"} Nov 01 09:02:39.387010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:39.386991 5710 scope.go:122] "RemoveContainer" containerID="f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094" Nov 01 09:02:39.389257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:39.389217 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:02:39.389347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:39.389328 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:02:39.389407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:39.389381 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:02:40.412910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:40.412878 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" event={"ID":"ba49da73-1f72-46fd-affa-a286031ead32","Type":"ContainerStarted","Data":"e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d"} Nov 01 09:02:41.187746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:41.185136 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:02:41.187746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:41.185280 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 09:02:41.187746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:41.185325 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 09:02:42.188749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:42.186205 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 09:02:42.188749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:42.186277 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 09:02:43.199468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:43.199429 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 09:02:43.200138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:43.200092 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 09:02:44.187219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:44.187179 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 09:02:44.187566 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:44.187533 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 09:02:45.198742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:45.194353 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 09:02:45.198742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:45.194424 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 09:02:45.478845 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:45.478749 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:02:46.184296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:46.184251 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 01 09:02:46.184495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:46.184335 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 01 09:02:47.207442 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:47.207401 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:02:47.208385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:47.208358 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:02:47.209755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:47.209690 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:02:49.826359 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:49.826322 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx"] Nov 01 09:02:49.928683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:49.928645 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm"] Nov 01 09:02:54.827061 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:54.826994 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" podUID="fac2844b-8082-4577-b118-e785d36a469b" containerName="payload-processing" containerID="cri-o://075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832" gracePeriod=25 Nov 01 09:02:54.929962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:54.929910 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" podUID="5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" containerName="payload-pre-processing" containerID="cri-o://dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9" gracePeriod=25 Nov 01 09:02:55.121803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.121768 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 09:02:55.166170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.166135 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j\" (UniqueName: \"kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j\") pod \"fac2844b-8082-4577-b118-e785d36a469b\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " Nov 01 09:02:55.166378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.166188 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume\") pod \"fac2844b-8082-4577-b118-e785d36a469b\" (UID: \"fac2844b-8082-4577-b118-e785d36a469b\") " Nov 01 09:02:55.166656 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.166620 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume" pod "fac2844b-8082-4577-b118-e785d36a469b" (UID: "fac2844b-8082-4577-b118-e785d36a469b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:02:55.167288 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.167262 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac2844b-8082-4577-b118-e785d36a469b-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:55.169421 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.169366 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j" pod "fac2844b-8082-4577-b118-e785d36a469b" (UID: "fac2844b-8082-4577-b118-e785d36a469b"). InnerVolumeSpecName "kube-api-access-bml2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:02:55.237007 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.236976 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 09:02:55.268900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.268853 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume\") pod \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " Nov 01 09:02:55.269168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.269143 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf\" (UniqueName: \"kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf\") pod \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\" (UID: \"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c\") " Nov 01 09:02:55.269365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.269335 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume" pod "5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" (UID: "5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:02:55.269881 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.269856 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bml2j\" (UniqueName: \"kubernetes.io/projected/fac2844b-8082-4577-b118-e785d36a469b-kube-api-access-bml2j\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:55.270014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.269884 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:55.272269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.272234 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf" pod "5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" (UID: "5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c"). InnerVolumeSpecName "kube-api-access-29xpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:02:55.370977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.370942 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xpf\" (UniqueName: \"kubernetes.io/projected/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c-kube-api-access-29xpf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:02:55.672417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.672321 5710 generic.go:356] "Generic (PLEG): container finished" podID="fac2844b-8082-4577-b118-e785d36a469b" containerID="075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832" exitCode=0 Nov 01 09:02:55.672417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.672390 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" Nov 01 09:02:55.672676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.672410 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" event={"ID":"fac2844b-8082-4577-b118-e785d36a469b","Type":"ContainerDied","Data":"075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832"} Nov 01 09:02:55.672676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.672505 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx" event={"ID":"fac2844b-8082-4577-b118-e785d36a469b","Type":"ContainerDied","Data":"128182e6245b47eed8768e53f1b39844a4155796b93f3768624fc105637e8453"} Nov 01 09:02:55.672676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.672536 5710 scope.go:122] "RemoveContainer" containerID="075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832" Nov 01 09:02:55.674441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.674413 5710 generic.go:356] "Generic (PLEG): container finished" podID="5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" containerID="dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9" exitCode=0 Nov 01 09:02:55.674610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.674542 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" event={"ID":"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c","Type":"ContainerDied","Data":"dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9"} Nov 01 09:02:55.674610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.674590 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" event={"ID":"5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c","Type":"ContainerDied","Data":"f34c0a635b9ec61b03d414336bb5190f4b8cbea9baa8484aef17fb8bed0a99b3"} Nov 01 09:02:55.674696 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.674626 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm" Nov 01 09:02:55.685299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.685260 5710 scope.go:122] "RemoveContainer" containerID="075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832" Nov 01 09:02:55.685834 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:55.685799 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832\": container with ID starting with 075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832 not found: ID does not exist" containerID="075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832" Nov 01 09:02:55.685931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.685853 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832"} err="failed to get container status \"075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832\": rpc error: code = NotFound desc = could not find container \"075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832\": container with ID starting with 075fa682c352b75b3a391d51202d69af31b86cee847d2858cc90a180b0fad832 not found: ID does not exist" Nov 01 09:02:55.685931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.685883 5710 scope.go:122] "RemoveContainer" containerID="dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9" Nov 01 09:02:55.701638 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.701610 5710 scope.go:122] "RemoveContainer" containerID="dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9" Nov 01 09:02:55.702039 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:02:55.702016 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9\": container with ID starting with dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9 not found: ID does not exist" containerID="dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9" Nov 01 09:02:55.702109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.702049 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9"} err="failed to get container status \"dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9\": rpc error: code = NotFound desc = could not find container \"dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9\": container with ID starting with dff69f07e41b14b1f794ecef3af77223c21488d10650a7e67f0ffc8fe144abb9 not found: ID does not exist" Nov 01 09:02:55.827292 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.827252 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm"] Nov 01 09:02:55.873798 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.873765 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3901b8-6954d8b97f-mm7rm"] Nov 01 09:02:55.937126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.937039 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx"] Nov 01 09:02:55.942871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:55.942841 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3901b8-558886c749-p9fjx"] Nov 01 09:02:56.683289 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:56.683255 5710 generic.go:356] "Generic (PLEG): container finished" podID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerID="1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4" exitCode=0 Nov 01 09:02:56.683565 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:56.683426 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" event={"ID":"bac2d1a7-18ca-4294-b82c-4607fcad47d0","Type":"ContainerDied","Data":"1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4"} Nov 01 09:02:56.684031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:56.683999 5710 scope.go:122] "RemoveContainer" containerID="1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4" Nov 01 09:02:56.685879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:56.685845 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:02:56.685994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:56.685932 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:02:56.685994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:56.685978 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:02:57.059562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:57.059479 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" path="/var/lib/kubelet/pods/5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c/volumes" Nov 01 09:02:57.060039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:57.060022 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac2844b-8082-4577-b118-e785d36a469b" path="/var/lib/kubelet/pods/fac2844b-8082-4577-b118-e785d36a469b/volumes" Nov 01 09:02:57.716446 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:57.716410 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" event={"ID":"bac2d1a7-18ca-4294-b82c-4607fcad47d0","Type":"ContainerStarted","Data":"cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f"} Nov 01 09:02:58.374777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:58.374748 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:02:58.633480 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:58.633386 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:02:58.634573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:58.634549 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 01 09:02:58.634704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:58.634614 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 01 09:02:59.632659 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:59.632622 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 01 09:02:59.633191 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:02:59.632694 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 01 09:03:00.633358 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:00.633311 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 01 09:03:00.633915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:00.633374 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 01 09:03:00.866478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:00.866455 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:03:00.875171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:00.875149 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:03:01.036521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:01.036423 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l"] Nov 01 09:03:01.135572 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:01.135530 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn"] Nov 01 09:03:01.638753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:01.638261 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 01 09:03:01.638753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:01.638366 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 01 09:03:02.642739 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:02.636517 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 01 09:03:02.642739 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:02.636578 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 01 09:03:03.637657 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:03.637625 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:03:03.638768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:03.638749 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:03:06.037288 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.037197 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" podUID="bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" containerName="payload-processing" containerID="cri-o://fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc" gracePeriod=25 Nov 01 09:03:06.135955 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.135914 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" podUID="d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" containerName="payload-pre-processing" containerID="cri-o://4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150" gracePeriod=25 Nov 01 09:03:06.322628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.322504 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:03:06.429921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.429887 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:03:06.438329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.435264 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk\" (UniqueName: \"kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk\") pod \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " Nov 01 09:03:06.438329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.437358 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume\") pod \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\" (UID: \"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230\") " Nov 01 09:03:06.438817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.438777 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume" pod "bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" (UID: "bb8d6382-3aaf-41e9-a3dd-cf3b70a08230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:06.442657 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.442624 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk" pod "bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" (UID: "bb8d6382-3aaf-41e9-a3dd-cf3b70a08230"). InnerVolumeSpecName "kube-api-access-bswjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:06.538653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.538615 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f\" (UniqueName: \"kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f\") pod \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " Nov 01 09:03:06.538946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.538928 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume\") pod \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\" (UID: \"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2\") " Nov 01 09:03:06.539310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.539283 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume" pod "d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" (UID: "d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:06.539476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.539458 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswjk\" (UniqueName: \"kubernetes.io/projected/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-kube-api-access-bswjk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:06.539544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.539482 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:06.539544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.539497 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:06.540992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.540948 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f" pod "d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" (UID: "d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2"). InnerVolumeSpecName "kube-api-access-75k7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:06.640482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.640440 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75k7f\" (UniqueName: \"kubernetes.io/projected/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2-kube-api-access-75k7f\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:06.818691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.818650 5710 generic.go:356] "Generic (PLEG): container finished" podID="bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" containerID="fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc" exitCode=0 Nov 01 09:03:06.818925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.818695 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" event={"ID":"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230","Type":"ContainerDied","Data":"fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc"} Nov 01 09:03:06.818925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.818761 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" Nov 01 09:03:06.818925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.818791 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l" event={"ID":"bb8d6382-3aaf-41e9-a3dd-cf3b70a08230","Type":"ContainerDied","Data":"0130343ec05214e939ade6a169ac0749c9554e07177c8c3c689d5f958f22e3e4"} Nov 01 09:03:06.818925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.818816 5710 scope.go:122] "RemoveContainer" containerID="fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc" Nov 01 09:03:06.820383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.820360 5710 generic.go:356] "Generic (PLEG): container finished" podID="d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" containerID="4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150" exitCode=0 Nov 01 09:03:06.820490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.820408 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" Nov 01 09:03:06.820490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.820437 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" event={"ID":"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2","Type":"ContainerDied","Data":"4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150"} Nov 01 09:03:06.820575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.820491 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn" event={"ID":"d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2","Type":"ContainerDied","Data":"b5f19140ced1ab141ec9edec16bfd3b324ebce9f40f6209aa604256b07aa9b5b"} Nov 01 09:03:06.831044 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.831025 5710 scope.go:122] "RemoveContainer" containerID="fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc" Nov 01 09:03:06.831458 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:06.831434 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc\": container with ID starting with fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc not found: ID does not exist" containerID="fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc" Nov 01 09:03:06.831583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.831470 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc"} err="failed to get container status \"fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc\": rpc error: code = NotFound desc = could not find container \"fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc\": container with ID starting with fcbbe2f8c7f6cc2677e5a9def74dfb1362ccf57578c4fc86dbadafe9ee4e87cc not found: ID does not exist" Nov 01 09:03:06.831583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.831497 5710 scope.go:122] "RemoveContainer" containerID="4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150" Nov 01 09:03:06.841579 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.841557 5710 scope.go:122] "RemoveContainer" containerID="4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150" Nov 01 09:03:06.841923 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:06.841902 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150\": container with ID starting with 4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150 not found: ID does not exist" containerID="4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150" Nov 01 09:03:06.842011 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.841937 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150"} err="failed to get container status \"4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150\": rpc error: code = NotFound desc = could not find container \"4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150\": container with ID starting with 4eaf4c304a4fa26c7479b3de741bd2d42334b7f0a26b02a9b5447b4b0fde4150 not found: ID does not exist" Nov 01 09:03:06.973706 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:06.973617 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn"] Nov 01 09:03:07.020020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:07.019990 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1f453-79bc9b657f-lnltn"] Nov 01 09:03:07.059135 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:07.059107 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" path="/var/lib/kubelet/pods/d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2/volumes" Nov 01 09:03:07.082445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:07.082409 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l"] Nov 01 09:03:07.089108 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:07.089076 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1f453-5d74c6cd97-s9m9l"] Nov 01 09:03:09.060395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:09.060364 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" path="/var/lib/kubelet/pods/bb8d6382-3aaf-41e9-a3dd-cf3b70a08230/volumes" Nov 01 09:03:11.486378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:11.486344 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:03:14.894095 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:14.894063 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx"] Nov 01 09:03:14.894573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:14.894353 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" podUID="646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" containerName="maas-api" containerID="cri-o://f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e" gracePeriod=30 Nov 01 09:03:15.238704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.238675 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 09:03:15.336745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.325528 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk\" (UniqueName: \"kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk\") pod \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " Nov 01 09:03:15.336745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.325575 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls\") pod \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " Nov 01 09:03:15.336745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.325748 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs\") pod \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " Nov 01 09:03:15.336745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.325926 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca\") pod \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\" (UID: \"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b\") " Nov 01 09:03:15.363760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.363409 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca" pod "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" (UID: "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:15.366785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.364521 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls" pod "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" (UID: "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:03:15.369633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.369595 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs" pod "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" (UID: "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:03:15.369960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.369934 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk" pod "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" (UID: "646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b"). InnerVolumeSpecName "kube-api-access-xz9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:15.428597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.428557 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:15.428597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.428596 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:15.428889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.428610 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9wk\" (UniqueName: \"kubernetes.io/projected/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-kube-api-access-xz9wk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:15.428889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.428626 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:15.483373 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:15.483277 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:03:15.771028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.770937 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz"] Nov 01 09:03:15.890405 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.890374 5710 generic.go:356] "Generic (PLEG): container finished" podID="646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" containerID="f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e" exitCode=0 Nov 01 09:03:15.890574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.890423 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" Nov 01 09:03:15.890574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.890430 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" event={"ID":"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b","Type":"ContainerDied","Data":"f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e"} Nov 01 09:03:15.890574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.890474 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx" event={"ID":"646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b","Type":"ContainerDied","Data":"9157195d6a0ef58c0052ad965cbb76d34bec9fca75b693c0c35e3ded40b96785"} Nov 01 09:03:15.890574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.890497 5710 scope.go:122] "RemoveContainer" containerID="f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e" Nov 01 09:03:15.896197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.896169 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz"] Nov 01 09:03:15.902847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.902675 5710 scope.go:122] "RemoveContainer" containerID="f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e" Nov 01 09:03:15.903055 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:15.903035 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e\": container with ID starting with f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e not found: ID does not exist" containerID="f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e" Nov 01 09:03:15.903106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:15.903067 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e"} err="failed to get container status \"f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e\": rpc error: code = NotFound desc = could not find container \"f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e\": container with ID starting with f36da72d4a1787c3b0d458479394c19032eee8274d4a1900afb9b80225e75a8e not found: ID does not exist" Nov 01 09:03:16.217361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:16.217201 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx"] Nov 01 09:03:16.224867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:16.224834 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3901b8-56698f4944-l4rgx"] Nov 01 09:03:17.061831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:17.061795 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" path="/var/lib/kubelet/pods/646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b/volumes" Nov 01 09:03:20.771968 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:20.771922 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" podUID="e637f02a-2a9b-4272-97ed-2af0b6b72080" containerName="payload-processing" containerID="cri-o://8c7f625224749ad3adaa8e01c3b2259764b2280888510b6ce1b0d896451e2cdf" gracePeriod=25 Nov 01 09:03:20.897495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:20.897453 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" podUID="c88930cb-d255-4b18-a73f-e4c28560a8d1" containerName="payload-pre-processing" containerID="cri-o://fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449" gracePeriod=25 Nov 01 09:03:20.930347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:20.930307 5710 generic.go:356] "Generic (PLEG): container finished" podID="e637f02a-2a9b-4272-97ed-2af0b6b72080" containerID="8c7f625224749ad3adaa8e01c3b2259764b2280888510b6ce1b0d896451e2cdf" exitCode=0 Nov 01 09:03:20.930517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:20.930373 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" event={"ID":"e637f02a-2a9b-4272-97ed-2af0b6b72080","Type":"ContainerDied","Data":"8c7f625224749ad3adaa8e01c3b2259764b2280888510b6ce1b0d896451e2cdf"} Nov 01 09:03:21.054413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.054384 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:03:21.126605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.126551 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume\") pod \"e637f02a-2a9b-4272-97ed-2af0b6b72080\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " Nov 01 09:03:21.126791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.126630 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx\" (UniqueName: \"kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx\") pod \"e637f02a-2a9b-4272-97ed-2af0b6b72080\" (UID: \"e637f02a-2a9b-4272-97ed-2af0b6b72080\") " Nov 01 09:03:21.127360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.127314 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume" pod "e637f02a-2a9b-4272-97ed-2af0b6b72080" (UID: "e637f02a-2a9b-4272-97ed-2af0b6b72080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:21.130393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.130366 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx" pod "e637f02a-2a9b-4272-97ed-2af0b6b72080" (UID: "e637f02a-2a9b-4272-97ed-2af0b6b72080"). InnerVolumeSpecName "kube-api-access-9rrnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:21.206587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.206560 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:03:21.228491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.228451 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrnx\" (UniqueName: \"kubernetes.io/projected/e637f02a-2a9b-4272-97ed-2af0b6b72080-kube-api-access-9rrnx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:21.228491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.228484 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e637f02a-2a9b-4272-97ed-2af0b6b72080-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:21.335797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.335002 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume\") pod \"c88930cb-d255-4b18-a73f-e4c28560a8d1\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " Nov 01 09:03:21.335797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.335109 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx\" (UniqueName: \"kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx\") pod \"c88930cb-d255-4b18-a73f-e4c28560a8d1\" (UID: \"c88930cb-d255-4b18-a73f-e4c28560a8d1\") " Nov 01 09:03:21.341740 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.337458 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume" pod "c88930cb-d255-4b18-a73f-e4c28560a8d1" (UID: "c88930cb-d255-4b18-a73f-e4c28560a8d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:21.355186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.355147 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx" pod "c88930cb-d255-4b18-a73f-e4c28560a8d1" (UID: "c88930cb-d255-4b18-a73f-e4c28560a8d1"). InnerVolumeSpecName "kube-api-access-9zrbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:21.439001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.438962 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88930cb-d255-4b18-a73f-e4c28560a8d1-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:21.439239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.439212 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrbx\" (UniqueName: \"kubernetes.io/projected/c88930cb-d255-4b18-a73f-e4c28560a8d1-kube-api-access-9zrbx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:21.939564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.939530 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" Nov 01 09:03:21.939564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.939545 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz" event={"ID":"e637f02a-2a9b-4272-97ed-2af0b6b72080","Type":"ContainerDied","Data":"eacfc597513c839faac4fae4bd55f0d22e1017d6fbdd25edddcabcc39c2f735d"} Nov 01 09:03:21.940122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.939613 5710 scope.go:122] "RemoveContainer" containerID="8c7f625224749ad3adaa8e01c3b2259764b2280888510b6ce1b0d896451e2cdf" Nov 01 09:03:21.941177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.941151 5710 generic.go:356] "Generic (PLEG): container finished" podID="c88930cb-d255-4b18-a73f-e4c28560a8d1" containerID="fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449" exitCode=0 Nov 01 09:03:21.941290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.941203 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" Nov 01 09:03:21.941290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.941233 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" event={"ID":"c88930cb-d255-4b18-a73f-e4c28560a8d1","Type":"ContainerDied","Data":"fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449"} Nov 01 09:03:21.941376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.941288 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz" event={"ID":"c88930cb-d255-4b18-a73f-e4c28560a8d1","Type":"ContainerDied","Data":"4ba3362dc9c5637666de55b2baf36c67ceff723fdc3e14ce0dba1cad46dfbe0c"} Nov 01 09:03:21.950417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.950396 5710 scope.go:122] "RemoveContainer" containerID="fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449" Nov 01 09:03:21.960002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.959976 5710 scope.go:122] "RemoveContainer" containerID="fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449" Nov 01 09:03:21.960339 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:21.960317 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449\": container with ID starting with fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449 not found: ID does not exist" containerID="fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449" Nov 01 09:03:21.960408 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:21.960347 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449"} err="failed to get container status \"fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449\": rpc error: code = NotFound desc = could not find container \"fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449\": container with ID starting with fa2f63b93fdcaf26d62ce4471b50f006edadedee3907e8d32d05a78069828449 not found: ID does not exist" Nov 01 09:03:22.094557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:22.094529 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz"] Nov 01 09:03:22.140845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:22.140817 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-7bb9f6f4dc-dh9qz"] Nov 01 09:03:22.200808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:22.200749 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz"] Nov 01 09:03:22.205289 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:22.205254 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73660a-b47c4bc87-7kzfz"] Nov 01 09:03:23.059683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:23.059644 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88930cb-d255-4b18-a73f-e4c28560a8d1" path="/var/lib/kubelet/pods/c88930cb-d255-4b18-a73f-e4c28560a8d1/volumes" Nov 01 09:03:23.060168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:23.060152 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e637f02a-2a9b-4272-97ed-2af0b6b72080" path="/var/lib/kubelet/pods/e637f02a-2a9b-4272-97ed-2af0b6b72080/volumes" Nov 01 09:03:23.967019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:23.966988 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2"] Nov 01 09:03:23.967252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:23.967213 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" containerID="cri-o://6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36" gracePeriod=30 Nov 01 09:03:29.241306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.241245 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 09:03:29.378001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.377959 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378190 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378039 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378190 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378129 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378270 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378322 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378354 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378376 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:29.378614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378441 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378485 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378520 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket\") pod \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\" (UID: \"d25050d6-773f-4e20-81ef-7f8f0ab796ed\") " Nov 01 09:03:29.378862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.378837 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:29.379120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.379077 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:29.379120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.379109 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.379324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.379131 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.379324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.379231 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:29.379442 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.379385 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:29.381770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.381683 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:03:29.381770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.381687 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:29.381770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.381749 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:29.382004 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.381802 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv" pod "d25050d6-773f-4e20-81ef-7f8f0ab796ed" (UID: "d25050d6-773f-4e20-81ef-7f8f0ab796ed"). InnerVolumeSpecName "kube-api-access-sh2fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481009 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481042 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481053 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481062 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481071 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2fv\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-kube-api-access-sh2fv\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481082 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25050d6-773f-4e20-81ef-7f8f0ab796ed-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:29.481049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:29.481091 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25050d6-773f-4e20-81ef-7f8f0ab796ed-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:30.003023 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.002991 5710 generic.go:356] "Generic (PLEG): container finished" podID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerID="6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36" exitCode=0 Nov 01 09:03:30.003252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.003043 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" Nov 01 09:03:30.003252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.003088 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" event={"ID":"d25050d6-773f-4e20-81ef-7f8f0ab796ed","Type":"ContainerDied","Data":"6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36"} Nov 01 09:03:30.003252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.003145 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2" event={"ID":"d25050d6-773f-4e20-81ef-7f8f0ab796ed","Type":"ContainerDied","Data":"55bfb261f0fda3d03f937fe9fd97703b1f5d5abb4df5d5b8e6457a03a99ba7cb"} Nov 01 09:03:30.003252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.003167 5710 scope.go:122] "RemoveContainer" containerID="6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36" Nov 01 09:03:30.014382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.014360 5710 scope.go:122] "RemoveContainer" containerID="833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969" Nov 01 09:03:30.023914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.023890 5710 scope.go:122] "RemoveContainer" containerID="6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36" Nov 01 09:03:30.024221 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:30.024202 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36\": container with ID starting with 6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36 not found: ID does not exist" containerID="6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36" Nov 01 09:03:30.024279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.024231 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36"} err="failed to get container status \"6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36\": rpc error: code = NotFound desc = could not find container \"6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36\": container with ID starting with 6ee2431bee1a07fe546e7a311aa6245ae1613d9b50cb28bb67772394b0f38c36 not found: ID does not exist" Nov 01 09:03:30.024279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.024252 5710 scope.go:122] "RemoveContainer" containerID="833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969" Nov 01 09:03:30.024501 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:30.024486 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969\": container with ID starting with 833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969 not found: ID does not exist" containerID="833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969" Nov 01 09:03:30.024550 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.024507 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969"} err="failed to get container status \"833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969\": rpc error: code = NotFound desc = could not find container \"833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969\": container with ID starting with 833798aeff7c3dd3c6346a2ddeef8fc1b081a49381aaeba9ebeff2ed025ec969 not found: ID does not exist" Nov 01 09:03:30.205929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.205892 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2"] Nov 01 09:03:30.253170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:30.253089 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d31acd45-openshift-default-6f6c789d9-xdvk2"] Nov 01 09:03:31.059777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:31.059740 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" path="/var/lib/kubelet/pods/d25050d6-773f-4e20-81ef-7f8f0ab796ed/volumes" Nov 01 09:03:31.964703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:31.964661 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4"] Nov 01 09:03:32.063184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:32.063142 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x"] Nov 01 09:03:36.965464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:36.965384 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" podUID="7e3f4430-ab28-4e09-b8c6-7381eba07e6b" containerName="payload-processing" containerID="cri-o://e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5" gracePeriod=25 Nov 01 09:03:37.064199 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.064160 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" podUID="6dd26f75-477a-4ea4-a64c-d059820935c4" containerName="payload-pre-processing" containerID="cri-o://e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a" gracePeriod=25 Nov 01 09:03:37.242055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.242026 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:03:37.343074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.343047 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:03:37.430301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.430256 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr\" (UniqueName: \"kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr\") pod \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " Nov 01 09:03:37.430517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.430439 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume\") pod \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\" (UID: \"7e3f4430-ab28-4e09-b8c6-7381eba07e6b\") " Nov 01 09:03:37.430886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.430858 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume" pod "7e3f4430-ab28-4e09-b8c6-7381eba07e6b" (UID: "7e3f4430-ab28-4e09-b8c6-7381eba07e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:37.431076 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.431059 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:37.432754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.432731 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr" pod "7e3f4430-ab28-4e09-b8c6-7381eba07e6b" (UID: "7e3f4430-ab28-4e09-b8c6-7381eba07e6b"). InnerVolumeSpecName "kube-api-access-g2jgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:37.532105 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.532009 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume\") pod \"6dd26f75-477a-4ea4-a64c-d059820935c4\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " Nov 01 09:03:37.532105 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.532051 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667\" (UniqueName: \"kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667\") pod \"6dd26f75-477a-4ea4-a64c-d059820935c4\" (UID: \"6dd26f75-477a-4ea4-a64c-d059820935c4\") " Nov 01 09:03:37.532419 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.532393 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume" pod "6dd26f75-477a-4ea4-a64c-d059820935c4" (UID: "6dd26f75-477a-4ea4-a64c-d059820935c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:37.532596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.532582 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd26f75-477a-4ea4-a64c-d059820935c4-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:37.532643 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.532599 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jgr\" (UniqueName: \"kubernetes.io/projected/7e3f4430-ab28-4e09-b8c6-7381eba07e6b-kube-api-access-g2jgr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:37.534412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.534388 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667" pod "6dd26f75-477a-4ea4-a64c-d059820935c4" (UID: "6dd26f75-477a-4ea4-a64c-d059820935c4"). InnerVolumeSpecName "kube-api-access-69667". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:37.633794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:37.633760 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69667\" (UniqueName: \"kubernetes.io/projected/6dd26f75-477a-4ea4-a64c-d059820935c4-kube-api-access-69667\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:38.079310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.079268 5710 generic.go:356] "Generic (PLEG): container finished" podID="7e3f4430-ab28-4e09-b8c6-7381eba07e6b" containerID="e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5" exitCode=0 Nov 01 09:03:38.079776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.079322 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" Nov 01 09:03:38.079776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.079353 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" event={"ID":"7e3f4430-ab28-4e09-b8c6-7381eba07e6b","Type":"ContainerDied","Data":"e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5"} Nov 01 09:03:38.079776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.079416 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4" event={"ID":"7e3f4430-ab28-4e09-b8c6-7381eba07e6b","Type":"ContainerDied","Data":"c6c8794be063ae06c25cc980720715b7ebf2b6e1fb8f55009d51628e7e9fc10f"} Nov 01 09:03:38.079776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.079441 5710 scope.go:122] "RemoveContainer" containerID="e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5" Nov 01 09:03:38.081680 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.081658 5710 generic.go:356] "Generic (PLEG): container finished" podID="6dd26f75-477a-4ea4-a64c-d059820935c4" containerID="e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a" exitCode=0 Nov 01 09:03:38.081906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.081765 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" event={"ID":"6dd26f75-477a-4ea4-a64c-d059820935c4","Type":"ContainerDied","Data":"e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a"} Nov 01 09:03:38.081906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.081804 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" Nov 01 09:03:38.081906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.081817 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x" event={"ID":"6dd26f75-477a-4ea4-a64c-d059820935c4","Type":"ContainerDied","Data":"278908e11ed95e8dcf2e6aca577f91e1bdc42112d55ab31708bd1bbf04620e29"} Nov 01 09:03:38.092994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.092970 5710 scope.go:122] "RemoveContainer" containerID="e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5" Nov 01 09:03:38.093326 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:38.093307 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5\": container with ID starting with e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5 not found: ID does not exist" containerID="e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5" Nov 01 09:03:38.093379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.093336 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5"} err="failed to get container status \"e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5\": rpc error: code = NotFound desc = could not find container \"e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5\": container with ID starting with e7b6e54c7106dcb7b809b1f9917bfac5f88a70fcbccc098b0791e8e88d6206d5 not found: ID does not exist" Nov 01 09:03:38.093379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.093354 5710 scope.go:122] "RemoveContainer" containerID="e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a" Nov 01 09:03:38.103019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.102996 5710 scope.go:122] "RemoveContainer" containerID="e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a" Nov 01 09:03:38.103371 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:38.103345 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a\": container with ID starting with e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a not found: ID does not exist" containerID="e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a" Nov 01 09:03:38.103444 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.103378 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a"} err="failed to get container status \"e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a\": rpc error: code = NotFound desc = could not find container \"e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a\": container with ID starting with e96f56bbf59fe6ed46a37bb570c9e2a69ba7466526f4d46ed50966aed54ed06a not found: ID does not exist" Nov 01 09:03:38.234801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.234692 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4"] Nov 01 09:03:38.281286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.281247 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-568f63-59c56c5c9c-km5w4"] Nov 01 09:03:38.384766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.384730 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x"] Nov 01 09:03:38.389131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:38.389100 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-568f63-c4864cd87-qcj2x"] Nov 01 09:03:39.058978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:39.058946 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd26f75-477a-4ea4-a64c-d059820935c4" path="/var/lib/kubelet/pods/6dd26f75-477a-4ea4-a64c-d059820935c4/volumes" Nov 01 09:03:39.059375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:39.059361 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3f4430-ab28-4e09-b8c6-7381eba07e6b" path="/var/lib/kubelet/pods/7e3f4430-ab28-4e09-b8c6-7381eba07e6b/volumes" Nov 01 09:03:43.920310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:43.919387 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96"] Nov 01 09:03:43.920310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:43.919749 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" podUID="15cb4d9b-f16c-483c-8949-a8fecf0926f8" containerName="maas-api" containerID="cri-o://3a1fd8ac7307fa28f00f38997d2989169aa6ac9622d09862e4abd805cb057b03" gracePeriod=30 Nov 01 09:03:44.125005 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.124975 5710 generic.go:356] "Generic (PLEG): container finished" podID="15cb4d9b-f16c-483c-8949-a8fecf0926f8" containerID="3a1fd8ac7307fa28f00f38997d2989169aa6ac9622d09862e4abd805cb057b03" exitCode=0 Nov 01 09:03:44.125191 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.125045 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" event={"ID":"15cb4d9b-f16c-483c-8949-a8fecf0926f8","Type":"ContainerDied","Data":"3a1fd8ac7307fa28f00f38997d2989169aa6ac9622d09862e4abd805cb057b03"} Nov 01 09:03:44.389758 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.389708 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:03:44.424992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.424957 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs\") pod \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " Nov 01 09:03:44.425210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.425052 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd\" (UniqueName: \"kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd\") pod \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " Nov 01 09:03:44.425210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.425184 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\") pod \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " Nov 01 09:03:44.425334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.425212 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca\") pod \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\" (UID: \"15cb4d9b-f16c-483c-8949-a8fecf0926f8\") " Nov 01 09:03:44.425707 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.425666 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca" pod "15cb4d9b-f16c-483c-8949-a8fecf0926f8" (UID: "15cb4d9b-f16c-483c-8949-a8fecf0926f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:44.427361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.427335 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs" pod "15cb4d9b-f16c-483c-8949-a8fecf0926f8" (UID: "15cb4d9b-f16c-483c-8949-a8fecf0926f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:03:44.427472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.427356 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls" pod "15cb4d9b-f16c-483c-8949-a8fecf0926f8" (UID: "15cb4d9b-f16c-483c-8949-a8fecf0926f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:03:44.427472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.427424 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd" pod "15cb4d9b-f16c-483c-8949-a8fecf0926f8" (UID: "15cb4d9b-f16c-483c-8949-a8fecf0926f8"). InnerVolumeSpecName "kube-api-access-ktgzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:44.526617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.526581 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktgzd\" (UniqueName: \"kubernetes.io/projected/15cb4d9b-f16c-483c-8949-a8fecf0926f8-kube-api-access-ktgzd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:44.526617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.526616 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:44.526617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.526626 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15cb4d9b-f16c-483c-8949-a8fecf0926f8-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:44.526916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:44.526635 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cb4d9b-f16c-483c-8949-a8fecf0926f8-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:45.133785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:45.133713 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" Nov 01 09:03:45.134343 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:45.133756 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96" event={"ID":"15cb4d9b-f16c-483c-8949-a8fecf0926f8","Type":"ContainerDied","Data":"aa711e68ed4a5165949223daa9f2f4e49b09e355b2b618523a693fa62f863192"} Nov 01 09:03:45.134343 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:45.133902 5710 scope.go:122] "RemoveContainer" containerID="3a1fd8ac7307fa28f00f38997d2989169aa6ac9622d09862e4abd805cb057b03" Nov 01 09:03:45.292149 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:45.292048 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96"] Nov 01 09:03:45.337130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:45.337090 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1f453-585fd65b4d-jvz96"] Nov 01 09:03:45.477246 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:45.477194 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:03:47.063270 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:47.063233 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cb4d9b-f16c-483c-8949-a8fecf0926f8" path="/var/lib/kubelet/pods/15cb4d9b-f16c-483c-8949-a8fecf0926f8/volumes" Nov 01 09:03:51.665978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:51.665948 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv"] Nov 01 09:03:51.666393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:51.666181 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" containerID="cri-o://7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80" gracePeriod=30 Nov 01 09:03:53.186058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186027 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr"] Nov 01 09:03:53.186649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186634 5710 cpu_manager.go:414] "containerMap: removing container" podUID="5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" containerName="payload-pre-processing" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186651 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" containerName="payload-pre-processing" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186666 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c88930cb-d255-4b18-a73f-e4c28560a8d1" containerName="payload-pre-processing" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186671 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88930cb-d255-4b18-a73f-e4c28560a8d1" containerName="payload-pre-processing" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186682 5710 cpu_manager.go:414] "containerMap: removing container" podUID="646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" containerName="maas-api" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186687 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" containerName="maas-api" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186698 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ddef433d-68ce-4ef7-a1f2-772a7db251e1" containerName="payload-pre-processing" Nov 01 09:03:53.186703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186703 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddef433d-68ce-4ef7-a1f2-772a7db251e1" containerName="payload-pre-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186731 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e637f02a-2a9b-4272-97ed-2af0b6b72080" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186738 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e637f02a-2a9b-4272-97ed-2af0b6b72080" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186753 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186758 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186766 5710 cpu_manager.go:414] "containerMap: removing container" podUID="fac2844b-8082-4577-b118-e785d36a469b" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186771 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac2844b-8082-4577-b118-e785d36a469b" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186779 5710 cpu_manager.go:414] "containerMap: removing container" podUID="58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186783 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186793 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" containerName="payload-pre-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186797 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" containerName="payload-pre-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186806 5710 cpu_manager.go:414] "containerMap: removing container" podUID="7e3f4430-ab28-4e09-b8c6-7381eba07e6b" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186810 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3f4430-ab28-4e09-b8c6-7381eba07e6b" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186818 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186832 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186839 5710 cpu_manager.go:414] "containerMap: removing container" podUID="bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186843 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" containerName="payload-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186853 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6dd26f75-477a-4ea4-a64c-d059820935c4" containerName="payload-pre-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186858 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd26f75-477a-4ea4-a64c-d059820935c4" containerName="payload-pre-processing" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186867 5710 cpu_manager.go:414] "containerMap: removing container" podUID="15cb4d9b-f16c-483c-8949-a8fecf0926f8" containerName="maas-api" Nov 01 09:03:53.186939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186871 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cb4d9b-f16c-483c-8949-a8fecf0926f8" containerName="maas-api" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186955 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac2844b-8082-4577-b118-e785d36a469b" containerName="payload-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186963 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="58585a6d-4a10-49dd-ab74-8d0d3cac2fb6" containerName="payload-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186971 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ecc876-4aad-480c-b5fa-ac4a2d0aacc2" containerName="payload-pre-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186976 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3f4430-ab28-4e09-b8c6-7381eba07e6b" containerName="payload-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186983 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186989 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8d6382-3aaf-41e9-a3dd-cf3b70a08230" containerName="payload-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.186994 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd26f75-477a-4ea4-a64c-d059820935c4" containerName="payload-pre-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.187001 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cb4d9b-f16c-483c-8949-a8fecf0926f8" containerName="maas-api" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.187010 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcd841e-5e8c-4df1-9d22-7ab1624d3c8c" containerName="payload-pre-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.187014 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88930cb-d255-4b18-a73f-e4c28560a8d1" containerName="payload-pre-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.187022 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="646ffb4c-22dd-4e93-b4c9-5ed44f35fe1b" containerName="maas-api" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.187033 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddef433d-68ce-4ef7-a1f2-772a7db251e1" containerName="payload-pre-processing" Nov 01 09:03:53.187514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.187040 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e637f02a-2a9b-4272-97ed-2af0b6b72080" containerName="payload-processing" Nov 01 09:03:53.191994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.191973 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.242953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.242899 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-095909-openshift-default-dockercfg-n424h\"" Nov 01 09:03:53.243990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.243955 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.248982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.248944 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.249289 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.249268 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.250213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.250189 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.250449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.250432 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.250634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.250619 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.250832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.250811 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk68d\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.262753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.251080 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.264248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.264215 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.392745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.392344 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr"] Nov 01 09:03:53.423162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.423117 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424007 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424231 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424215 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424372 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424482 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424601 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424752 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.424896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.424882 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk68d\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.425092 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.425074 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.428215 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.425492 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.428402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.423858 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.428518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.425757 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.428637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.426711 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.428851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.426957 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.449461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.449385 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.495148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.495109 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.506332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.506301 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.506499 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.506364 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk68d\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d\") pod \"e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:53.528785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:53.528752 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:54.005409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.005365 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr"] Nov 01 09:03:54.012266 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:03:54.012236 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd31e0d3_924d_400c_bb26_fccdcfd388b0.slice/crio-cbfd28849031983a1befc266b9e914359ff95ac65cf135c3bb645d16ccf9ac4b WatchSource:0}: Error finding container cbfd28849031983a1befc266b9e914359ff95ac65cf135c3bb645d16ccf9ac4b: Status 404 returned error can't find the container with id cbfd28849031983a1befc266b9e914359ff95ac65cf135c3bb645d16ccf9ac4b Nov 01 09:03:54.014827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.014794 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:03:54.014939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.014863 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:03:54.014939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.014891 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:03:54.194096 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.194052 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" event={"ID":"bd31e0d3-924d-400c-bb26-fccdcfd388b0","Type":"ContainerStarted","Data":"02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d"} Nov 01 09:03:54.194532 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.194115 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" event={"ID":"bd31e0d3-924d-400c-bb26-fccdcfd388b0","Type":"ContainerStarted","Data":"cbfd28849031983a1befc266b9e914359ff95ac65cf135c3bb645d16ccf9ac4b"} Nov 01 09:03:54.310865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.310753 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podStartSLOduration=1.3107331549999999 podStartE2EDuration="1.310733155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:03:54.307530896 +0000 UTC m=+1857.395867567" watchObservedRunningTime="2026-11-01 09:03:54.310733155 +0000 UTC m=+1857.399069824" Nov 01 09:03:54.529364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:54.529328 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:55.603757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:55.602909 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:56.210122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:56.210089 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:56.211138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:56.211117 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:03:56.952056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:56.952027 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:03:57.006024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.005992 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006207 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006107 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006207 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006147 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006291 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:57.006495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006474 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006566 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006538 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006476 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:57.006630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006580 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006645 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006670 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:57.006753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006698 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy\") pod \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\" (UID: \"9ac479b6-2bb8-410e-aca6-d3791e9f8744\") " Nov 01 09:03:57.006753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006741 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:03:57.007009 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.006982 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:57.007660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.007558 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.007660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.007583 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.007660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.007598 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.007660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.007611 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.007660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.007627 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.009334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.009296 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:03:57.009711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.009542 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "kube-api-access-ct8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:57.009711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.009670 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:03:57.010058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.010038 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token" pod "9ac479b6-2bb8-410e-aca6-d3791e9f8744" (UID: "9ac479b6-2bb8-410e-aca6-d3791e9f8744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:03:57.107158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.107121 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5"] Nov 01 09:03:57.107829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.107802 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" Nov 01 09:03:57.107829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.107825 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" Nov 01 09:03:57.108014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.107871 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" Nov 01 09:03:57.108014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.107877 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" Nov 01 09:03:57.108014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.108004 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25050d6-773f-4e20-81ef-7f8f0ab796ed" containerName="istio-proxy" Nov 01 09:03:57.108014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.108012 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" Nov 01 09:03:57.108150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.108026 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerName="istio-proxy" Nov 01 09:03:57.109029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.109001 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.109079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.109034 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-kube-api-access-ct8ln\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.109079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.109049 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.109079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.109061 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac479b6-2bb8-410e-aca6-d3791e9f8744-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:03:57.117044 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.116871 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.166253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.166205 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73660a-dockercfg-hc2cb\"" Nov 01 09:03:57.166440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.166316 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5"] Nov 01 09:03:57.166440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.166331 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73660a\"" Nov 01 09:03:57.210441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.210342 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqlm\" (UniqueName: \"kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm\") pod \"payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.210613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.210511 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.218676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.218647 5710 generic.go:356] "Generic (PLEG): container finished" podID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" containerID="7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80" exitCode=0 Nov 01 09:03:57.218854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.218704 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" Nov 01 09:03:57.218854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.218741 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" event={"ID":"9ac479b6-2bb8-410e-aca6-d3791e9f8744","Type":"ContainerDied","Data":"7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80"} Nov 01 09:03:57.218854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.218797 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv" event={"ID":"9ac479b6-2bb8-410e-aca6-d3791e9f8744","Type":"ContainerDied","Data":"2f5ff12aba86d89dfe8955a9e7f74d8379f4036ba4b1c7afa7481de33e5fdc5d"} Nov 01 09:03:57.218854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.218820 5710 scope.go:122] "RemoveContainer" containerID="7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80" Nov 01 09:03:57.229441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.229421 5710 scope.go:122] "RemoveContainer" containerID="ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd" Nov 01 09:03:57.239310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.239291 5710 scope.go:122] "RemoveContainer" containerID="7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80" Nov 01 09:03:57.239625 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:57.239605 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80\": container with ID starting with 7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80 not found: ID does not exist" containerID="7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80" Nov 01 09:03:57.239678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.239637 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80"} err="failed to get container status \"7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80\": rpc error: code = NotFound desc = could not find container \"7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80\": container with ID starting with 7837fdd7cd504e05c125ec2237a65064198bcbca2fef9add846c76f363adcd80 not found: ID does not exist" Nov 01 09:03:57.239678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.239655 5710 scope.go:122] "RemoveContainer" containerID="ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd" Nov 01 09:03:57.239949 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:03:57.239928 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd\": container with ID starting with ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd not found: ID does not exist" containerID="ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd" Nov 01 09:03:57.240004 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.239957 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd"} err="failed to get container status \"ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd\": rpc error: code = NotFound desc = could not find container \"ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd\": container with ID starting with ef4d530a832fe72bb231372beda6f66c102fd9a627631b8e1ece8cbd666664dd not found: ID does not exist" Nov 01 09:03:57.312531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.312488 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.313392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.313199 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqlm\" (UniqueName: \"kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm\") pod \"payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.313606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.313589 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.370877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.370847 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqlm\" (UniqueName: \"kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm\") pod \"payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.382159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.382129 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv"] Nov 01 09:03:57.387549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.387522 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c1f453-openshift-default-5bfd6758c5-rs8mv"] Nov 01 09:03:57.434989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.434953 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:03:57.819113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:57.819031 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5"] Nov 01 09:03:57.825929 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:03:57.825888 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28af8d2e_8809_430f_a082_006b0087b4fe.slice/crio-71ede97e0a330f4ddbea3521c61650bc649b4e1e1100c318bcca7efc54b91c44 WatchSource:0}: Error finding container 71ede97e0a330f4ddbea3521c61650bc649b4e1e1100c318bcca7efc54b91c44: Status 404 returned error can't find the container with id 71ede97e0a330f4ddbea3521c61650bc649b4e1e1100c318bcca7efc54b91c44 Nov 01 09:03:58.227729 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:58.227672 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" event={"ID":"28af8d2e-8809-430f-a082-006b0087b4fe","Type":"ContainerStarted","Data":"71ede97e0a330f4ddbea3521c61650bc649b4e1e1100c318bcca7efc54b91c44"} Nov 01 09:03:59.059236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.059191 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac479b6-2bb8-410e-aca6-d3791e9f8744" path="/var/lib/kubelet/pods/9ac479b6-2bb8-410e-aca6-d3791e9f8744/volumes" Nov 01 09:03:59.274631 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.274583 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l"] Nov 01 09:03:59.275156 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.274853 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" containerID="cri-o://62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee" gracePeriod=30 Nov 01 09:03:59.278305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.278275 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w"] Nov 01 09:03:59.278552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.278517 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" containerID="cri-o://0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680" gracePeriod=30 Nov 01 09:03:59.321012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.320933 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh"] Nov 01 09:03:59.321181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.321148 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" containerID="cri-o://53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63" gracePeriod=30 Nov 01 09:03:59.323581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.323555 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z"] Nov 01 09:03:59.323776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:03:59.323754 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" containerID="cri-o://7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b" gracePeriod=30 Nov 01 09:04:00.578918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:00.576465 5710 patch_prober.go:29] interesting pod/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:04:00.578918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:00.576546 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:04:01.025226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.025186 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v"] Nov 01 09:04:01.039375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.034317 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v"] Nov 01 09:04:01.039375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.034535 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.084125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.084087 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-095909\"" Nov 01 09:04:01.084333 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.084280 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-095909-dockercfg-5jz64\"" Nov 01 09:04:01.123973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.123931 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.124176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.124040 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjst\" (UniqueName: \"kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst\") pod \"payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.225407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.225372 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.225588 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.225452 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjst\" (UniqueName: \"kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst\") pod \"payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.226159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.226137 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.283923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.283842 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjst\" (UniqueName: \"kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst\") pod \"payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.365704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.365674 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:01.741331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:01.741295 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v"] Nov 01 09:04:01.747930 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:01.747898 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dfbc06_96bb_4fe4_881c_110ffe5095e5.slice/crio-674eff148e636af59d4555a7328a2ad4e7f746ee21e1f211e4b44f8ae402ac55 WatchSource:0}: Error finding container 674eff148e636af59d4555a7328a2ad4e7f746ee21e1f211e4b44f8ae402ac55: Status 404 returned error can't find the container with id 674eff148e636af59d4555a7328a2ad4e7f746ee21e1f211e4b44f8ae402ac55 Nov 01 09:04:02.256518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:02.256473 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" event={"ID":"86dfbc06-96bb-4fe4-881c-110ffe5095e5","Type":"ContainerStarted","Data":"acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632"} Nov 01 09:04:02.256785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:02.256569 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" event={"ID":"86dfbc06-96bb-4fe4-881c-110ffe5095e5","Type":"ContainerStarted","Data":"674eff148e636af59d4555a7328a2ad4e7f746ee21e1f211e4b44f8ae402ac55"} Nov 01 09:04:02.256785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:02.256689 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:02.366769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:02.366686 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" podStartSLOduration=2.366669705 podStartE2EDuration="2.366669705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:02.364906317 +0000 UTC m=+1865.453242991" watchObservedRunningTime="2026-11-01 09:04:02.366669705 +0000 UTC m=+1865.455006378" Nov 01 09:04:04.274782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.274740 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" event={"ID":"28af8d2e-8809-430f-a082-006b0087b4fe","Type":"ContainerStarted","Data":"915c38cc2a470e6a9ef12cdc47a5094dda198818962618168700ca44bde35bbd"} Nov 01 09:04:04.275186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.274846 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:04:04.385186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.385126 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" podStartSLOduration=1.998207575 podStartE2EDuration="7.385109407s" totalImagesPullingTime="5.386901832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:04.383605292 +0000 UTC m=+1867.471941960" watchObservedRunningTime="2026-11-01 09:04:04.385109407 +0000 UTC m=+1867.473446078" Nov 01 09:04:04.720360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.720323 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 09:04:04.756028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.755980 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 09:04:04.808479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.808455 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 09:04:04.808818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.808795 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.808910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.808894 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.808953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.808926 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.808976 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.808978 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.809115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809023 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809059 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809087 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809123 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809164 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809194 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809250 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809286 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809362 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809390 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809438 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809481 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809641 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data\") pod \"e798ef27-c3a9-4502-b6d6-257386db3b28\" (UID: \"e798ef27-c3a9-4502-b6d6-257386db3b28\") " Nov 01 09:04:04.809690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809684 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket\") pod \"86173b76-196a-4c9a-8e77-387714bb7ce6\" (UID: \"86173b76-196a-4c9a-8e77-387714bb7ce6\") " Nov 01 09:04:04.809833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809690 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.809992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.809949 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.810087 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810046 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.810273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810254 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.810459 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810440 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.810564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810494 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.810564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810498 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:04:04.810564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810538 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.810778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810575 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.810778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810604 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.810778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810618 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.810778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810631 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.810778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810645 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.810778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810660 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.811001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.810847 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:04:04.812275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.812247 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:04:04.814463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.814226 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.814463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.814235 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:04:04.814463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.814324 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "kube-api-access-fhpbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.814463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.814432 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.814463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.814453 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token" pod "86173b76-196a-4c9a-8e77-387714bb7ce6" (UID: "86173b76-196a-4c9a-8e77-387714bb7ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.815791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.815769 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "kube-api-access-5b6qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.815992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.815970 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy" pod "e798ef27-c3a9-4502-b6d6-257386db3b28" (UID: "e798ef27-c3a9-4502-b6d6-257386db3b28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.819325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.819301 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 09:04:04.912175 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912117 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912357 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912190 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912357 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912246 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912357 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912306 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912357 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912349 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912414 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.912599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912444 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.912599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912496 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912533 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912598 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.912843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912682 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.912843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912704 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.912843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912767 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.912843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912816 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.913103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912862 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.913103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912869 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.913103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.912959 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert\") pod \"8028b953-7c08-4b94-9832-9c979607609d\" (UID: \"8028b953-7c08-4b94-9832-9c979607609d\") " Nov 01 09:04:04.913103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913073 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.913320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913128 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.913320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913195 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.913320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913235 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert\") pod \"b5a024d8-ba46-40b6-8490-9475eca60e5d\" (UID: \"b5a024d8-ba46-40b6-8490-9475eca60e5d\") " Nov 01 09:04:04.913320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913266 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.913320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913276 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.913946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913912 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.913946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913941 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913957 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913973 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.913988 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e798ef27-c3a9-4502-b6d6-257386db3b28-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914002 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86173b76-196a-4c9a-8e77-387714bb7ce6-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914017 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914032 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhpbq\" (UniqueName: \"kubernetes.io/projected/86173b76-196a-4c9a-8e77-387714bb7ce6-kube-api-access-fhpbq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914046 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914060 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914074 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914088 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914103 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6qs\" (UniqueName: \"kubernetes.io/projected/e798ef27-c3a9-4502-b6d6-257386db3b28-kube-api-access-5b6qs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914117 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86173b76-196a-4c9a-8e77-387714bb7ce6-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914130 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e798ef27-c3a9-4502-b6d6-257386db3b28-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914893 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914148 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914893 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914162 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:04.914893 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.914509 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:04:04.915381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.915350 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:04:04.915630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.915603 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.915630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.915613 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.915914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.915888 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.916005 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.915980 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:04:04.916093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.916072 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.916156 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.916118 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.916611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.916586 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:04.916763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.916642 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "kube-api-access-zxmqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.917121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.917096 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx" pod "8028b953-7c08-4b94-9832-9c979607609d" (UID: "8028b953-7c08-4b94-9832-9c979607609d"). InnerVolumeSpecName "kube-api-access-x27nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.917188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.917176 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:04.917831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:04.917810 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo" pod "b5a024d8-ba46-40b6-8490-9475eca60e5d" (UID: "b5a024d8-ba46-40b6-8490-9475eca60e5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:04:05.015261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015215 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8028b953-7c08-4b94-9832-9c979607609d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015260 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015270 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015279 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a024d8-ba46-40b6-8490-9475eca60e5d-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015289 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015298 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8028b953-7c08-4b94-9832-9c979607609d-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015307 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8028b953-7c08-4b94-9832-9c979607609d-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015315 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015323 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x27nx\" (UniqueName: \"kubernetes.io/projected/8028b953-7c08-4b94-9832-9c979607609d-kube-api-access-x27nx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015333 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015342 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmqs\" (UniqueName: \"kubernetes.io/projected/b5a024d8-ba46-40b6-8490-9475eca60e5d-kube-api-access-zxmqs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015350 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.015520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.015358 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a024d8-ba46-40b6-8490-9475eca60e5d-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:05.283381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.283340 5710 generic.go:356] "Generic (PLEG): container finished" podID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerID="0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680" exitCode=0 Nov 01 09:04:05.283909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.283386 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" Nov 01 09:04:05.283909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.283420 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" event={"ID":"86173b76-196a-4c9a-8e77-387714bb7ce6","Type":"ContainerDied","Data":"0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680"} Nov 01 09:04:05.283909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.283488 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w" event={"ID":"86173b76-196a-4c9a-8e77-387714bb7ce6","Type":"ContainerDied","Data":"4fef676e38c848d7c499eaf086f4021784d3ed88765a315268ff6f8dbdca7f17"} Nov 01 09:04:05.283909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.283522 5710 scope.go:122] "RemoveContainer" containerID="0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680" Nov 01 09:04:05.285160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.285123 5710 generic.go:356] "Generic (PLEG): container finished" podID="8028b953-7c08-4b94-9832-9c979607609d" containerID="7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b" exitCode=0 Nov 01 09:04:05.285300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.285217 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" Nov 01 09:04:05.285300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.285241 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" event={"ID":"8028b953-7c08-4b94-9832-9c979607609d","Type":"ContainerDied","Data":"7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b"} Nov 01 09:04:05.285300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.285281 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z" event={"ID":"8028b953-7c08-4b94-9832-9c979607609d","Type":"ContainerDied","Data":"b759afcff1649b144b9db0bad19fa54f01b6c2fa5ae58b24aa7b4f8e44b0c46c"} Nov 01 09:04:05.287334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.286959 5710 generic.go:356] "Generic (PLEG): container finished" podID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerID="53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63" exitCode=0 Nov 01 09:04:05.287334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.287010 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" Nov 01 09:04:05.287334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.287029 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" event={"ID":"e798ef27-c3a9-4502-b6d6-257386db3b28","Type":"ContainerDied","Data":"53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63"} Nov 01 09:04:05.287334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.287071 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh" event={"ID":"e798ef27-c3a9-4502-b6d6-257386db3b28","Type":"ContainerDied","Data":"8ac3fa535500c8d131cb61ffa23c1cb1265dd96cd68a080cad999e401c8ef198"} Nov 01 09:04:05.298239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.289191 5710 generic.go:356] "Generic (PLEG): container finished" podID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerID="62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee" exitCode=0 Nov 01 09:04:05.298239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.290100 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" Nov 01 09:04:05.298239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.290618 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" event={"ID":"b5a024d8-ba46-40b6-8490-9475eca60e5d","Type":"ContainerDied","Data":"62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee"} Nov 01 09:04:05.298239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.290668 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l" event={"ID":"b5a024d8-ba46-40b6-8490-9475eca60e5d","Type":"ContainerDied","Data":"be588b49d1e17a68510a5a70e425999416333f59c858beb8e991c3d96243159c"} Nov 01 09:04:05.356854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.356821 5710 scope.go:122] "RemoveContainer" containerID="0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680" Nov 01 09:04:05.357790 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:05.357751 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680\": container with ID starting with 0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680 not found: ID does not exist" containerID="0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680" Nov 01 09:04:05.357908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.357806 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680"} err="failed to get container status \"0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680\": rpc error: code = NotFound desc = could not find container \"0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680\": container with ID starting with 0c8ba839f208268c2d4844c8b725aa742e04e8609650697e9712bbc1e91b5680 not found: ID does not exist" Nov 01 09:04:05.357908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.357832 5710 scope.go:122] "RemoveContainer" containerID="7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b" Nov 01 09:04:05.368596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.368574 5710 scope.go:122] "RemoveContainer" containerID="7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b" Nov 01 09:04:05.368914 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:05.368895 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b\": container with ID starting with 7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b not found: ID does not exist" containerID="7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b" Nov 01 09:04:05.368965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.368923 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b"} err="failed to get container status \"7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b\": rpc error: code = NotFound desc = could not find container \"7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b\": container with ID starting with 7d541eb81d908f3e097a3251c6fd9426e84ba2fb85eef4940bf1c3f6ffe1521b not found: ID does not exist" Nov 01 09:04:05.368965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.368944 5710 scope.go:122] "RemoveContainer" containerID="53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63" Nov 01 09:04:05.379176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.379151 5710 scope.go:122] "RemoveContainer" containerID="53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63" Nov 01 09:04:05.379555 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:05.379531 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63\": container with ID starting with 53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63 not found: ID does not exist" containerID="53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63" Nov 01 09:04:05.379623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.379568 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63"} err="failed to get container status \"53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63\": rpc error: code = NotFound desc = could not find container \"53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63\": container with ID starting with 53b52b899ae647a1ea45a729196736e8359a8b5392bbcc79bc3f3cbb2009ff63 not found: ID does not exist" Nov 01 09:04:05.379623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.379593 5710 scope.go:122] "RemoveContainer" containerID="62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee" Nov 01 09:04:05.391634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.391611 5710 scope.go:122] "RemoveContainer" containerID="62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee" Nov 01 09:04:05.392016 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:05.391994 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee\": container with ID starting with 62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee not found: ID does not exist" containerID="62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee" Nov 01 09:04:05.392076 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.392028 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee"} err="failed to get container status \"62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee\": rpc error: code = NotFound desc = could not find container \"62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee\": container with ID starting with 62dfb1820ac51ab4f6f872f1177edebefefd4ab0290e3e264ac3b7b30083efee not found: ID does not exist" Nov 01 09:04:05.592273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.592226 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh"] Nov 01 09:04:05.600985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.600947 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jjkkh"] Nov 01 09:04:05.703363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.703278 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z"] Nov 01 09:04:05.708089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.708058 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-nhf8z"] Nov 01 09:04:05.809938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.809893 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w"] Nov 01 09:04:05.814200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.814169 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-jwf7w"] Nov 01 09:04:05.917491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.917453 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l"] Nov 01 09:04:05.921756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:05.921708 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-751a73-openshift-default-5c98fdbd5-4bc4l"] Nov 01 09:04:06.350792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:06.350747 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:04:07.059683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:07.059650 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8028b953-7c08-4b94-9832-9c979607609d" path="/var/lib/kubelet/pods/8028b953-7c08-4b94-9832-9c979607609d/volumes" Nov 01 09:04:07.060211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:07.060195 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" path="/var/lib/kubelet/pods/86173b76-196a-4c9a-8e77-387714bb7ce6/volumes" Nov 01 09:04:07.060669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:07.060653 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" path="/var/lib/kubelet/pods/b5a024d8-ba46-40b6-8490-9475eca60e5d/volumes" Nov 01 09:04:07.061442 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:07.061427 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" path="/var/lib/kubelet/pods/e798ef27-c3a9-4502-b6d6-257386db3b28/volumes" Nov 01 09:04:13.291352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.287968 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:04:13.875811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.875777 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd"] Nov 01 09:04:13.876403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876387 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" Nov 01 09:04:13.876460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876405 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" Nov 01 09:04:13.876460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876429 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" Nov 01 09:04:13.876460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876434 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" Nov 01 09:04:13.876460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876443 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" Nov 01 09:04:13.876460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876447 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" Nov 01 09:04:13.876460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876461 5710 cpu_manager.go:414] "containerMap: removing container" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" Nov 01 09:04:13.876655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876466 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" Nov 01 09:04:13.876655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876604 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e798ef27-c3a9-4502-b6d6-257386db3b28" containerName="istio-proxy" Nov 01 09:04:13.876655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876622 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a024d8-ba46-40b6-8490-9475eca60e5d" containerName="istio-proxy" Nov 01 09:04:13.876655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876631 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="86173b76-196a-4c9a-8e77-387714bb7ce6" containerName="istio-proxy" Nov 01 09:04:13.876655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.876644 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8028b953-7c08-4b94-9832-9c979607609d" containerName="istio-proxy" Nov 01 09:04:13.881677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.881656 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:13.914747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.914694 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:13.914966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.914940 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d789f\" (UniqueName: \"kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f\") pod \"payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:13.934674 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.934646 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd"] Nov 01 09:04:13.977509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.977479 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-991c28\"" Nov 01 09:04:13.977694 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:13.977536 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-991c28-dockercfg-qlkkq\"" Nov 01 09:04:14.016692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.016654 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:14.016878 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.016810 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d789f\" (UniqueName: \"kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f\") pod \"payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:14.017413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.017392 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:14.074811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.074782 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d789f\" (UniqueName: \"kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f\") pod \"payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:14.196465 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.196371 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:14.577672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.577590 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd"] Nov 01 09:04:14.582692 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:14.582661 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae79c83_4f36_45ec_af2f_d792fdd9e9e2.slice/crio-3c17b3c106eb8f2662c77e26638101d5fd6951120ffc5e88ecf5670d18066946 WatchSource:0}: Error finding container 3c17b3c106eb8f2662c77e26638101d5fd6951120ffc5e88ecf5670d18066946: Status 404 returned error can't find the container with id 3c17b3c106eb8f2662c77e26638101d5fd6951120ffc5e88ecf5670d18066946 Nov 01 09:04:14.658753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.658684 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt"] Nov 01 09:04:14.662979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.662951 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.681912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.681866 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv"] Nov 01 09:04:14.682153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.682124 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" podUID="d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" containerName="maas-api" containerID="cri-o://cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b" gracePeriod=30 Nov 01 09:04:14.718320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.718285 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt"] Nov 01 09:04:14.733692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.733652 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtrh\" (UniqueName: \"kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh\") pod \"payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.735094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.734937 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.836958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.836843 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtrh\" (UniqueName: \"kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh\") pod \"payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.837109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.836990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.837970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.837945 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.973165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.973132 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtrh\" (UniqueName: \"kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh\") pod \"payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:14.978464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:14.978433 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:15.007585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.007558 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:04:15.070707 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.070612 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs\") pod \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " Nov 01 09:04:15.071059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.070994 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94\" (UniqueName: \"kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94\") pod \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " Nov 01 09:04:15.071302 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.071257 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\") pod \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " Nov 01 09:04:15.072422 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.071745 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca\") pod \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\" (UID: \"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059\") " Nov 01 09:04:15.073208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.073168 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca" pod "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" (UID: "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:04:15.135741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.134965 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs" pod "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" (UID: "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:04:15.163763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.162945 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls" pod "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" (UID: "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:04:15.188763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.186416 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:15.188763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.186443 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:15.188763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.186455 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:15.188763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.186923 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94" pod "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" (UID: "d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059"). InnerVolumeSpecName "kube-api-access-fxq94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:15.291483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.291140 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxq94\" (UniqueName: \"kubernetes.io/projected/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059-kube-api-access-fxq94\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:15.294932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.294899 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:04:15.406977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.406629 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt"] Nov 01 09:04:15.411449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.411411 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" event={"ID":"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2","Type":"ContainerStarted","Data":"d3d6ad1cd9540a11749522794c4fa9b4dedd1d11e752b4321628fa6b8fdf2323"} Nov 01 09:04:15.411606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.411487 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:15.411606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.411509 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" event={"ID":"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2","Type":"ContainerStarted","Data":"3c17b3c106eb8f2662c77e26638101d5fd6951120ffc5e88ecf5670d18066946"} Nov 01 09:04:15.414753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.413702 5710 generic.go:356] "Generic (PLEG): container finished" podID="d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" containerID="cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b" exitCode=0 Nov 01 09:04:15.414753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.413778 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" Nov 01 09:04:15.414753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.413883 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" event={"ID":"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059","Type":"ContainerDied","Data":"cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b"} Nov 01 09:04:15.414753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.413917 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv" event={"ID":"d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059","Type":"ContainerDied","Data":"4018ab80ee7952961410e8d0138d14010639b0a43039ef903df6f839b71f0e3e"} Nov 01 09:04:15.414753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.413945 5710 scope.go:122] "RemoveContainer" containerID="cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b" Nov 01 09:04:15.446587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.446557 5710 scope.go:122] "RemoveContainer" containerID="cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b" Nov 01 09:04:15.447077 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:15.447049 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b\": container with ID starting with cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b not found: ID does not exist" containerID="cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b" Nov 01 09:04:15.447182 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.447086 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b"} err="failed to get container status \"cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b\": rpc error: code = NotFound desc = could not find container \"cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b\": container with ID starting with cbbe87811feab80aaab9b2334806481cbe6b68d13b47582c28ae22ec8078141b not found: ID does not exist" Nov 01 09:04:15.496845 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:15.496801 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:04:15.521355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.521166 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" podStartSLOduration=2.5211458 podStartE2EDuration="2.5211458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:15.520211644 +0000 UTC m=+1878.608548317" watchObservedRunningTime="2026-11-01 09:04:15.5211458 +0000 UTC m=+1878.609482473" Nov 01 09:04:15.625025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.624990 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv"] Nov 01 09:04:15.629803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:15.629769 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-568f63-7bcf59fb76-wf2lv"] Nov 01 09:04:16.425054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:16.424998 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" event={"ID":"29ac004d-6065-4aeb-a587-344df49951b1","Type":"ContainerStarted","Data":"205fb41e59d6b34c75f6225799435717609cdb35265dcd3f6cb63092b6b01d8e"} Nov 01 09:04:16.425249 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:16.425072 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" event={"ID":"29ac004d-6065-4aeb-a587-344df49951b1","Type":"ContainerStarted","Data":"59cdeb0ccf640434709d1102528da4a344be4829252123ee3ab7563188b490cb"} Nov 01 09:04:16.425697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:16.425362 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:16.539355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:16.539298 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" podStartSLOduration=2.539283379 podStartE2EDuration="2.539283379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:16.53656323 +0000 UTC m=+1879.624899902" watchObservedRunningTime="2026-11-01 09:04:16.539283379 +0000 UTC m=+1879.627620050" Nov 01 09:04:17.059002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:17.058969 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" path="/var/lib/kubelet/pods/d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059/volumes" Nov 01 09:04:19.550525 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.550491 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk"] Nov 01 09:04:19.551120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.551105 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" containerName="maas-api" Nov 01 09:04:19.551166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.551121 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" containerName="maas-api" Nov 01 09:04:19.551266 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.551255 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45cbfa7-2fff-4563-ac5c-8a7e5d6e4059" containerName="maas-api" Nov 01 09:04:19.555029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.555005 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.640329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.640294 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eb28ec-openshift-default-dockercfg-bx8zp\"" Nov 01 09:04:19.654403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.654370 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk"] Nov 01 09:04:19.660151 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660120 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660348 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660193 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660348 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660286 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660347 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660493 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlbb\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660582 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660631 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660656 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.660770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.660754 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.762568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762526 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.762772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762626 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.762772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762655 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.762772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762702 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.763030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762858 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlbb\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.763030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762921 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.763030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762960 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.763030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.762985 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.763238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.763031 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.763888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.763862 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.767684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.767306 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.768206 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.767938 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.768798 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.768465 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.770690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.770666 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.771356 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.771332 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.787205 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.786978 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.824874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.824840 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlbb\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.828083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.828055 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token\") pod \"e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:19.901767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:19.893679 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:20.342575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.342537 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk"] Nov 01 09:04:20.349916 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:20.349884 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462e1c7c_0c60_4311_a492_de0082e58c32.slice/crio-0e8c8571ca84367d94f8ff1fa6cc560dfe1ac696ad5cd0df85ddb842b22a40d6 WatchSource:0}: Error finding container 0e8c8571ca84367d94f8ff1fa6cc560dfe1ac696ad5cd0df85ddb842b22a40d6: Status 404 returned error can't find the container with id 0e8c8571ca84367d94f8ff1fa6cc560dfe1ac696ad5cd0df85ddb842b22a40d6 Nov 01 09:04:20.352630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.352598 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:04:20.352759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.352668 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:04:20.352759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.352697 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:04:20.456081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.456037 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" event={"ID":"462e1c7c-0c60-4311-a492-de0082e58c32","Type":"ContainerStarted","Data":"036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f"} Nov 01 09:04:20.456282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.456114 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" event={"ID":"462e1c7c-0c60-4311-a492-de0082e58c32","Type":"ContainerStarted","Data":"0e8c8571ca84367d94f8ff1fa6cc560dfe1ac696ad5cd0df85ddb842b22a40d6"} Nov 01 09:04:20.572341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.572279 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podStartSLOduration=1.5722622849999999 podStartE2EDuration="1.572262285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:20.569908355 +0000 UTC m=+1883.658245038" watchObservedRunningTime="2026-11-01 09:04:20.572262285 +0000 UTC m=+1883.660598957" Nov 01 09:04:20.893915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:20.893875 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:21.899948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:21.899918 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:22.478467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:22.478438 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:22.479328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:22.479301 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:04:25.212890 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:25.212859 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj"] Nov 01 09:04:25.213301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:25.213065 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" containerID="cri-o://eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af" gracePeriod=30 Nov 01 09:04:26.428903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.428868 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:04:26.450498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.450456 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf"] Nov 01 09:04:26.455414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.455391 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.510370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.510336 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf"] Nov 01 09:04:26.523750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.523691 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-23fa20-serving-cert\"" Nov 01 09:04:26.555639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.555606 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.555898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.555666 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lznc\" (UniqueName: \"kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.555898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.555836 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.556012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.555929 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.657345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.657307 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.657571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.657401 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lznc\" (UniqueName: \"kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.657571 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:26.657467 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-serving-cert: secret "maas-api-e2e-ipp-b-23fa20-serving-cert" not found Nov 01 09:04:26.657571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.657488 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.657571 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:26.657557 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls podName:3f64cd8e-2d20-48cf-aa9f-11373510dd28 nodeName:}" failed. No retries permitted until 2026-11-01 09:04:27.157535915 +0000 UTC m=+1890.245872581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls") pod "maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" (UID: "3f64cd8e-2d20-48cf-aa9f-11373510dd28") : secret "maas-api-e2e-ipp-b-23fa20-serving-cert" not found Nov 01 09:04:26.657824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.657593 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.658295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.658273 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.660346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.660314 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:26.748818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:26.736477 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lznc\" (UniqueName: \"kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:27.039660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.039566 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp"] Nov 01 09:04:27.044400 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.044376 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.097501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.097467 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b893c8-openshift-default-dockercfg-5wbx6\"" Nov 01 09:04:27.100374 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.100338 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp"] Nov 01 09:04:27.178052 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178014 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178155 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178187 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178319 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178211 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4m8\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178407 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178439 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178588 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178560 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178670 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178639 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178709 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.178810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.178767 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:27.181398 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.181371 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:27.280107 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280069 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280188 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280219 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280251 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4m8\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280308 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280555 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280349 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280555 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280418 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280555 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280461 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280555 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280513 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280762 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.280878 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.280829 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.281057 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.281035 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.281127 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.281050 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.281127 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.281078 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.283102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.283078 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.283261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.283245 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.341580 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.341543 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4m8\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.341845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.341708 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token\") pod \"e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.360877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.360821 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:27.424489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.371682 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:27.523843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.523810 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4"] Nov 01 09:04:27.539245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.539193 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.540015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.539987 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4"] Nov 01 09:04:27.590750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.589475 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb28ec-dockercfg-z9rk7\"" Nov 01 09:04:27.590750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.589598 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb28ec\"" Nov 01 09:04:27.637854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.637667 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.637854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.637738 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jn6\" (UniqueName: \"kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6\") pod \"payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.739361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.739322 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jn6\" (UniqueName: \"kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6\") pod \"payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.739583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.739535 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.740324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.740299 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.797486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.797447 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jn6\" (UniqueName: \"kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6\") pod \"payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.866091 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.866055 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:27.885995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:27.885924 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf"] Nov 01 09:04:27.894040 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:27.893963 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f64cd8e_2d20_48cf_aa9f_11373510dd28.slice/crio-f46c574d2c4eb0b6d067af04df65a7379315dc4dd516ebdf903093d0e0de282b WatchSource:0}: Error finding container f46c574d2c4eb0b6d067af04df65a7379315dc4dd516ebdf903093d0e0de282b: Status 404 returned error can't find the container with id f46c574d2c4eb0b6d067af04df65a7379315dc4dd516ebdf903093d0e0de282b Nov 01 09:04:28.005405 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.005363 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp"] Nov 01 09:04:28.012304 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:28.012272 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eaedd8_27f5_4f7a_baf4_39c7f12ed3b3.slice/crio-6bf978ea7cf9631517bab0347ae7c44925dcb2a70ea3cea2b1a8bcbbf90ea5e4 WatchSource:0}: Error finding container 6bf978ea7cf9631517bab0347ae7c44925dcb2a70ea3cea2b1a8bcbbf90ea5e4: Status 404 returned error can't find the container with id 6bf978ea7cf9631517bab0347ae7c44925dcb2a70ea3cea2b1a8bcbbf90ea5e4 Nov 01 09:04:28.014927 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.014895 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:04:28.014987 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.014969 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:04:28.015023 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.014996 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:04:28.254366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.254284 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4"] Nov 01 09:04:28.261504 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:28.261474 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ef9834_8cd3_448b_a979_7bcb4481d4d7.slice/crio-03161284dea6caaf3e8053b3496c2a9480615d93f425c9082538a46bad7fe83c WatchSource:0}: Error finding container 03161284dea6caaf3e8053b3496c2a9480615d93f425c9082538a46bad7fe83c: Status 404 returned error can't find the container with id 03161284dea6caaf3e8053b3496c2a9480615d93f425c9082538a46bad7fe83c Nov 01 09:04:28.441489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.441447 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:04:28.598456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.598405 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" event={"ID":"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3","Type":"ContainerStarted","Data":"629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea"} Nov 01 09:04:28.599029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.598509 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" event={"ID":"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3","Type":"ContainerStarted","Data":"6bf978ea7cf9631517bab0347ae7c44925dcb2a70ea3cea2b1a8bcbbf90ea5e4"} Nov 01 09:04:28.600604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.600557 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" event={"ID":"58ef9834-8cd3-448b-a979-7bcb4481d4d7","Type":"ContainerStarted","Data":"e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37"} Nov 01 09:04:28.600782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.600615 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" event={"ID":"58ef9834-8cd3-448b-a979-7bcb4481d4d7","Type":"ContainerStarted","Data":"03161284dea6caaf3e8053b3496c2a9480615d93f425c9082538a46bad7fe83c"} Nov 01 09:04:28.602073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.602045 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" event={"ID":"3f64cd8e-2d20-48cf-aa9f-11373510dd28","Type":"ContainerStarted","Data":"f46c574d2c4eb0b6d067af04df65a7379315dc4dd516ebdf903093d0e0de282b"} Nov 01 09:04:28.716414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.716349 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podStartSLOduration=2.716330703 podStartE2EDuration="2.716330703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:28.714062354 +0000 UTC m=+1891.802399024" watchObservedRunningTime="2026-11-01 09:04:28.716330703 +0000 UTC m=+1891.804667376" Nov 01 09:04:28.837972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:28.834657 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" podStartSLOduration=1.8346355060000001 podStartE2EDuration="1.834635506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:28.832558049 +0000 UTC m=+1891.920894722" watchObservedRunningTime="2026-11-01 09:04:28.834635506 +0000 UTC m=+1891.922972179" Nov 01 09:04:29.361938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.361895 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:29.368159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.368127 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:29.611002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.610957 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" event={"ID":"3f64cd8e-2d20-48cf-aa9f-11373510dd28","Type":"ContainerStarted","Data":"c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880"} Nov 01 09:04:29.611487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.611397 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:29.611487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.611437 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:29.611487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.611462 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:29.612445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.612400 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:04:29.721060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:29.721009 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" podStartSLOduration=2.603277778 podStartE2EDuration="3.72099378s" totalImagesPullingTime="1.117716002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:29.720256666 +0000 UTC m=+1892.808593359" watchObservedRunningTime="2026-11-01 09:04:29.72099378 +0000 UTC m=+1892.809330451" Nov 01 09:04:30.486530 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.486465 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:04:30.596991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.596944 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597027 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597056 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597092 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597136 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597224 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597326 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:30.597403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597372 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597468 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597497 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f\") pod \"90022a98-79b2-4156-8ab1-3f7d82dd9490\" (UID: \"90022a98-79b2-4156-8ab1-3f7d82dd9490\") " Nov 01 09:04:30.597630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597524 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:04:30.597630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597531 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:30.597781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597704 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:30.597854 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.597834 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:30.598084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.598066 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90022a98-79b2-4156-8ab1-3f7d82dd9490-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.598154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.598090 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.598154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.598105 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.598154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.598120 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.598154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.598134 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.599914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.599884 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:30.600029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.599923 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "kube-api-access-s5z4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:04:30.600198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.600173 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:04:30.600323 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.600308 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo" pod "90022a98-79b2-4156-8ab1-3f7d82dd9490" (UID: "90022a98-79b2-4156-8ab1-3f7d82dd9490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:04:30.619759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.619701 5710 generic.go:356] "Generic (PLEG): container finished" podID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerID="eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af" exitCode=0 Nov 01 09:04:30.620224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.619769 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" Nov 01 09:04:30.620224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.619770 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" event={"ID":"90022a98-79b2-4156-8ab1-3f7d82dd9490","Type":"ContainerDied","Data":"eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af"} Nov 01 09:04:30.620224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.619842 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj" event={"ID":"90022a98-79b2-4156-8ab1-3f7d82dd9490","Type":"ContainerDied","Data":"29be1aac30e42471541b4a3ac4decf483af601c67b90a7f51e3b4075509faa25"} Nov 01 09:04:30.620224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.619876 5710 scope.go:122] "RemoveContainer" containerID="eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af" Nov 01 09:04:30.632925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.632902 5710 scope.go:122] "RemoveContainer" containerID="c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a" Nov 01 09:04:30.643507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.643477 5710 scope.go:122] "RemoveContainer" containerID="eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af" Nov 01 09:04:30.643970 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:30.643942 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af\": container with ID starting with eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af not found: ID does not exist" containerID="eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af" Nov 01 09:04:30.644077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.643974 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af"} err="failed to get container status \"eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af\": rpc error: code = NotFound desc = could not find container \"eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af\": container with ID starting with eeea30f3821c63c2f2fac00ab68a3b091e4cb045eb35f3833295ea04997c54af not found: ID does not exist" Nov 01 09:04:30.644077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.643993 5710 scope.go:122] "RemoveContainer" containerID="c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a" Nov 01 09:04:30.644274 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:30.644257 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a\": container with ID starting with c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a not found: ID does not exist" containerID="c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a" Nov 01 09:04:30.644329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.644281 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a"} err="failed to get container status \"c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a\": rpc error: code = NotFound desc = could not find container \"c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a\": container with ID starting with c1747bb5b0630f97faae3d91832f90674538d9fe0ff568b2f321895533d6f79a not found: ID does not exist" Nov 01 09:04:30.699853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.699812 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5z4f\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-kube-api-access-s5z4f\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.699853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.699845 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.699853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.699858 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.700168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.699871 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90022a98-79b2-4156-8ab1-3f7d82dd9490-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:04:30.752434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.752352 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj"] Nov 01 09:04:30.756259 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:30.756230 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-568f63-openshift-default-59f578746c-4zwrj"] Nov 01 09:04:31.075774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:31.075659 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" path="/var/lib/kubelet/pods/90022a98-79b2-4156-8ab1-3f7d82dd9490/volumes" Nov 01 09:04:35.625622 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:35.625542 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:04:36.297841 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.297801 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64"] Nov 01 09:04:36.298551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.298525 5710 cpu_manager.go:414] "containerMap: removing container" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" Nov 01 09:04:36.298551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.298548 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" Nov 01 09:04:36.298744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.298560 5710 cpu_manager.go:414] "containerMap: removing container" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" Nov 01 09:04:36.298744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.298565 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" Nov 01 09:04:36.298744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.298686 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" Nov 01 09:04:36.298744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.298694 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="90022a98-79b2-4156-8ab1-3f7d82dd9490" containerName="istio-proxy" Nov 01 09:04:36.305296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.305274 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.354228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.354194 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b893c8\"" Nov 01 09:04:36.354228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.354224 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b893c8-dockercfg-49mfv\"" Nov 01 09:04:36.356054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.356025 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64"] Nov 01 09:04:36.430163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.430073 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.430744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.430520 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhg86\" (UniqueName: \"kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.504752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.492161 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf"] Nov 01 09:04:36.504752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.503790 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.545579 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.544128 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.545579 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.544245 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhg86\" (UniqueName: \"kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.545579 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.545498 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.545973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.545938 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf"] Nov 01 09:04:36.602414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.602383 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhg86\" (UniqueName: \"kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.621188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.621146 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:36.646009 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.645974 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.646510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.646104 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfzd\" (UniqueName: \"kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd\") pod \"payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.747429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.747383 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.747653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.747505 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfzd\" (UniqueName: \"kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd\") pod \"payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.748152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.748131 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.805385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.805302 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfzd\" (UniqueName: \"kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd\") pod \"payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:36.860138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:36.860098 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:37.001768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.001700 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64"] Nov 01 09:04:37.008809 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:37.008782 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda72c59_82cd_48a0_a17b_e75a0289a8ad.slice/crio-dbe2c4f37a9d02153c96599639e015109d95822ca32263da0838451d631878a4 WatchSource:0}: Error finding container dbe2c4f37a9d02153c96599639e015109d95822ca32263da0838451d631878a4: Status 404 returned error can't find the container with id dbe2c4f37a9d02153c96599639e015109d95822ca32263da0838451d631878a4 Nov 01 09:04:37.240943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.240904 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf"] Nov 01 09:04:37.248374 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:37.248338 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2e28bb_3769_4a72_9463_7cb1b95667c6.slice/crio-330d20511c0e4cd2f118f03d0aab0d06e3fbe56b41529babd1ce6c3b027f5844 WatchSource:0}: Error finding container 330d20511c0e4cd2f118f03d0aab0d06e3fbe56b41529babd1ce6c3b027f5844: Status 404 returned error can't find the container with id 330d20511c0e4cd2f118f03d0aab0d06e3fbe56b41529babd1ce6c3b027f5844 Nov 01 09:04:37.688742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.685594 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" event={"ID":"1f2e28bb-3769-4a72-9463-7cb1b95667c6","Type":"ContainerStarted","Data":"26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027"} Nov 01 09:04:37.688742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.685666 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" event={"ID":"1f2e28bb-3769-4a72-9463-7cb1b95667c6","Type":"ContainerStarted","Data":"330d20511c0e4cd2f118f03d0aab0d06e3fbe56b41529babd1ce6c3b027f5844"} Nov 01 09:04:37.688742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.687928 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:37.700430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.699905 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" event={"ID":"cda72c59-82cd-48a0-a17b-e75a0289a8ad","Type":"ContainerStarted","Data":"9e559c54b2d4864e6f43cc507b0d6a254dd18cdfae2834d5cac08439179e0271"} Nov 01 09:04:37.700430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.699976 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" event={"ID":"cda72c59-82cd-48a0-a17b-e75a0289a8ad","Type":"ContainerStarted","Data":"dbe2c4f37a9d02153c96599639e015109d95822ca32263da0838451d631878a4"} Nov 01 09:04:37.700430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.700031 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:37.798512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.798454 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" podStartSLOduration=1.7984383 podStartE2EDuration="1.7984383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:37.796683585 +0000 UTC m=+1900.885020260" watchObservedRunningTime="2026-11-01 09:04:37.7984383 +0000 UTC m=+1900.886774972" Nov 01 09:04:37.944990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:37.921768 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" podStartSLOduration=1.921742812 podStartE2EDuration="1.921742812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:37.915385028 +0000 UTC m=+1901.003721784" watchObservedRunningTime="2026-11-01 09:04:37.921742812 +0000 UTC m=+1901.010079484" Nov 01 09:04:40.623893 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:40.623859 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:04:45.478742 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:45.478499 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:04:48.711035 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:48.710997 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:04:49.609909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.609867 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx"] Nov 01 09:04:49.614299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.614272 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.711895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.711853 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-095909-serving-cert\"" Nov 01 09:04:49.714941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.714917 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx"] Nov 01 09:04:49.717539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.717516 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:04:49.736641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.736607 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljm6z\" (UniqueName: \"kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.736802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.736756 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.736847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.736805 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.736883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.736865 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.837861 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.837824 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.838083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.837900 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.838083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.837942 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.838083 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:49.838026 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-serving-cert: secret "maas-api-e2e-ipp-a-095909-serving-cert" not found Nov 01 09:04:49.838253 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:04:49.838107 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls podName:9ea9f9c4-10f7-44c5-915e-c397521824ea nodeName:}" failed. No retries permitted until 2026-11-01 09:04:50.338085577 +0000 UTC m=+1913.426422245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls") pod "maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" (UID: "9ea9f9c4-10f7-44c5-915e-c397521824ea") : secret "maas-api-e2e-ipp-a-095909-serving-cert" not found Nov 01 09:04:49.838331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.838266 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljm6z\" (UniqueName: \"kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.838690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.838661 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.840856 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.840830 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:49.948807 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:49.941487 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljm6z\" (UniqueName: \"kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:50.351270 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:50.351231 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:50.353852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:50.353831 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-095909-5898fd89fd-485zx\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:50.530040 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:50.530007 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:50.920998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:50.920962 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx"] Nov 01 09:04:50.921851 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:04:50.921810 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea9f9c4_10f7_44c5_915e_c397521824ea.slice/crio-040e5800b894dae911786a1ee344cb869eb1d977b20aa39b438ee519c23283fb WatchSource:0}: Error finding container 040e5800b894dae911786a1ee344cb869eb1d977b20aa39b438ee519c23283fb: Status 404 returned error can't find the container with id 040e5800b894dae911786a1ee344cb869eb1d977b20aa39b438ee519c23283fb Nov 01 09:04:51.798730 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:51.798678 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" event={"ID":"9ea9f9c4-10f7-44c5-915e-c397521824ea","Type":"ContainerStarted","Data":"040e5800b894dae911786a1ee344cb869eb1d977b20aa39b438ee519c23283fb"} Nov 01 09:04:52.819687 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:52.819584 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" event={"ID":"9ea9f9c4-10f7-44c5-915e-c397521824ea","Type":"ContainerStarted","Data":"8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9"} Nov 01 09:04:52.821632 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:52.821594 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:04:52.938703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:52.938640 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" podStartSLOduration=3.047865139 podStartE2EDuration="3.938624168s" totalImagesPullingTime="890.759029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:04:52.936217667 +0000 UTC m=+1916.024554346" watchObservedRunningTime="2026-11-01 09:04:52.938624168 +0000 UTC m=+1916.026960869" Nov 01 09:04:59.859571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:04:59.859476 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:05:08.656358 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:08.656320 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:05:15.481426 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:15.481382 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:05:20.050488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.050435 5710 generic.go:356] "Generic (PLEG): container finished" podID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerID="02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d" exitCode=0 Nov 01 09:05:20.051054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.050517 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" event={"ID":"bd31e0d3-924d-400c-bb26-fccdcfd388b0","Type":"ContainerDied","Data":"02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d"} Nov 01 09:05:20.051054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.050994 5710 scope.go:122] "RemoveContainer" containerID="02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d" Nov 01 09:05:20.052640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.052613 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:20.052752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.052685 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:20.052752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.052734 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:20.541617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.541584 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876"] Nov 01 09:05:20.546176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.546150 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.596590 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.596116 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eb28ec-serving-cert\"" Nov 01 09:05:20.601050 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.601018 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876"] Nov 01 09:05:20.659508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.659459 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.659704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.659600 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.659704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.659632 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vzr\" (UniqueName: \"kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.659704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.659658 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.761190 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.761148 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.761391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.761240 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.761391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.761259 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vzr\" (UniqueName: \"kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.761391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.761274 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.761565 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:20.761543 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-serving-cert: secret "maas-api-e2e-ipp-b-eb28ec-serving-cert" not found Nov 01 09:05:20.761631 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:20.761621 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls podName:aa543cb9-c56e-436c-8610-1db7598247bf nodeName:}" failed. No retries permitted until 2026-11-01 09:05:21.261605145 +0000 UTC m=+1944.349941809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls") pod "maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" (UID: "aa543cb9-c56e-436c-8610-1db7598247bf") : secret "maas-api-e2e-ipp-b-eb28ec-serving-cert" not found Nov 01 09:05:20.761985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.761956 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.764064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.764035 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:20.835800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:20.833028 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vzr\" (UniqueName: \"kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:21.062215 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.062182 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" event={"ID":"bd31e0d3-924d-400c-bb26-fccdcfd388b0","Type":"ContainerStarted","Data":"a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920"} Nov 01 09:05:21.270263 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.270212 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:21.273120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.273085 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:21.461875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.461835 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:21.529290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.529195 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:05:21.529474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.529358 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 09:05:21.529474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.529412 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 09:05:21.871567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:21.871532 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876"] Nov 01 09:05:21.908747 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:05:21.907146 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa543cb9_c56e_436c_8610_1db7598247bf.slice/crio-c599a00a473d812d9167a6ee03928acc702462eab46f8b3deea07aa46964ac9a WatchSource:0}: Error finding container c599a00a473d812d9167a6ee03928acc702462eab46f8b3deea07aa46964ac9a: Status 404 returned error can't find the container with id c599a00a473d812d9167a6ee03928acc702462eab46f8b3deea07aa46964ac9a Nov 01 09:05:22.069558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:22.069520 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" event={"ID":"aa543cb9-c56e-436c-8610-1db7598247bf","Type":"ContainerStarted","Data":"c599a00a473d812d9167a6ee03928acc702462eab46f8b3deea07aa46964ac9a"} Nov 01 09:05:22.529763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:22.529712 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 09:05:22.529981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:22.529790 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 09:05:23.078538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:23.078501 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" event={"ID":"aa543cb9-c56e-436c-8610-1db7598247bf","Type":"ContainerStarted","Data":"76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34"} Nov 01 09:05:23.078976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:23.078650 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:23.207286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:23.207205 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" podStartSLOduration=2.27832059 podStartE2EDuration="3.207182519s" totalImagesPullingTime="928.861929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:05:23.206135819 +0000 UTC m=+1946.294472494" watchObservedRunningTime="2026-11-01 09:05:23.207182519 +0000 UTC m=+1946.295519191" Nov 01 09:05:23.529510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:23.529399 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 09:05:23.529510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:23.529467 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 09:05:24.529666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:24.529631 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 09:05:24.530106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:24.529691 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 09:05:25.529826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:25.529787 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 09:05:25.530255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:25.529855 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 09:05:26.210065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:26.210029 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:05:26.529923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:26.529836 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 01 09:05:26.529923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:26.529906 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 01 09:05:27.535883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:27.535848 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:05:27.537184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:27.537162 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:05:29.116196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:29.116168 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:05:30.336334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.336283 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg"] Nov 01 09:05:30.342039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.342009 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.426290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.426245 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72h8b\" (UniqueName: \"kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.426497 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.426301 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.426547 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.426525 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.426606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.426590 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.438610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.438569 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b893c8-serving-cert\"" Nov 01 09:05:30.442162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.442128 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg"] Nov 01 09:05:30.527528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.527493 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72h8b\" (UniqueName: \"kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.527829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.527551 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.527829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.527709 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.527829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.527791 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.528070 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:30.528028 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-serving-cert: secret "maas-api-e2e-ipp-a-b893c8-serving-cert" not found Nov 01 09:05:30.528195 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:30.528106 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls podName:94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1 nodeName:}" failed. No retries permitted until 2026-11-01 09:05:31.028089629 +0000 UTC m=+1954.116426292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls") pod "maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" (UID: "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1") : secret "maas-api-e2e-ipp-a-b893c8-serving-cert" not found Nov 01 09:05:30.528612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.528582 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.530931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.530895 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:30.588769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:30.588660 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72h8b\" (UniqueName: \"kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:31.046540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:31.046495 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:31.049332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:31.049294 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:31.256929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:31.256886 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:31.665340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:31.665303 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg"] Nov 01 09:05:31.672187 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:05:31.672155 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f4b186_c5eb_4d00_b9b8_c7a7bd63dfb1.slice/crio-135abe4155979252feb02bd277a3f55a000ce46335df30d05f3c5e2b0a2b5bd8 WatchSource:0}: Error finding container 135abe4155979252feb02bd277a3f55a000ce46335df30d05f3c5e2b0a2b5bd8: Status 404 returned error can't find the container with id 135abe4155979252feb02bd277a3f55a000ce46335df30d05f3c5e2b0a2b5bd8 Nov 01 09:05:31.861915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:31.861550 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d"] Nov 01 09:05:31.862227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:31.862198 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" containerID="cri-o://454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788" gracePeriod=30 Nov 01 09:05:32.165585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:32.165543 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" event={"ID":"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1","Type":"ContainerStarted","Data":"135abe4155979252feb02bd277a3f55a000ce46335df30d05f3c5e2b0a2b5bd8"} Nov 01 09:05:33.174558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:33.174500 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" event={"ID":"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1","Type":"ContainerStarted","Data":"f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d"} Nov 01 09:05:33.175012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:33.174582 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:33.289763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:33.289687 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" podStartSLOduration=2.312230981 podStartE2EDuration="3.289669878s" totalImagesPullingTime="977.438897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:05:33.286551059 +0000 UTC m=+1956.374887727" watchObservedRunningTime="2026-11-01 09:05:33.289669878 +0000 UTC m=+1956.378006549" Nov 01 09:05:36.302752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:36.302701 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:05:36.303159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:36.302786 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:05:37.164610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.164581 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 09:05:37.202706 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.202672 5710 generic.go:356] "Generic (PLEG): container finished" podID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerID="454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788" exitCode=0 Nov 01 09:05:37.202905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.202741 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" Nov 01 09:05:37.202905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.202752 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" event={"ID":"5b71a780-fc73-4bcf-979e-16bcd0ba5765","Type":"ContainerDied","Data":"454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788"} Nov 01 09:05:37.202905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.202804 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d" event={"ID":"5b71a780-fc73-4bcf-979e-16bcd0ba5765","Type":"ContainerDied","Data":"e11e2ffbf33dc75b1a361b12dd94908adca7128937e5d755250fe2e7f807ea06"} Nov 01 09:05:37.202905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.202829 5710 scope.go:122] "RemoveContainer" containerID="454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788" Nov 01 09:05:37.216164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.216139 5710 scope.go:122] "RemoveContainer" containerID="454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788" Nov 01 09:05:37.216568 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:37.216545 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788\": container with ID starting with 454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788 not found: ID does not exist" containerID="454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788" Nov 01 09:05:37.216677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.216577 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788"} err="failed to get container status \"454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788\": rpc error: code = NotFound desc = could not find container \"454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788\": container with ID starting with 454b7dd9cd7b5c90407d693e758890a9834298ce803e04c92cf6ede3bdde7788 not found: ID does not exist" Nov 01 09:05:37.244110 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244068 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244110 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244117 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244141 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244186 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244254 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244279 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244356 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244434 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244465 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket\") pod \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\" (UID: \"5b71a780-fc73-4bcf-979e-16bcd0ba5765\") " Nov 01 09:05:37.244681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244496 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:37.244681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244664 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:37.244942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244689 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:05:37.244942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.244820 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:37.245185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.245162 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:37.245352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.245332 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.245387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.245361 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.245387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.245377 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.245458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.245395 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.245458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.245427 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.246766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.246701 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:05:37.246857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.246736 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:05:37.247267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.247243 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:37.247267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.247249 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj" pod "5b71a780-fc73-4bcf-979e-16bcd0ba5765" (UID: "5b71a780-fc73-4bcf-979e-16bcd0ba5765"). InnerVolumeSpecName "kube-api-access-rd7mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:05:37.347079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.347033 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.347079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.347080 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd7mj\" (UniqueName: \"kubernetes.io/projected/5b71a780-fc73-4bcf-979e-16bcd0ba5765-kube-api-access-rd7mj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.347632 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.347097 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.347632 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.347111 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b71a780-fc73-4bcf-979e-16bcd0ba5765-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:37.618730 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.618676 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d"] Nov 01 09:05:37.621523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:37.621486 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6zk4d"] Nov 01 09:05:39.059538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:39.059500 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" path="/var/lib/kubelet/pods/5b71a780-fc73-4bcf-979e-16bcd0ba5765/volumes" Nov 01 09:05:39.187648 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:39.187619 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:05:39.560278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:39.560232 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl"] Nov 01 09:05:39.560534 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:39.560503 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" containerID="cri-o://79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203" gracePeriod=30 Nov 01 09:05:41.112065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:41.112019 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:05:41.112615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:41.112097 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:05:43.235384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:43.235338 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:05:44.966206 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:44.966176 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 09:05:45.058190 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058159 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058202 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058240 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058281 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058304 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058326 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058405 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058548 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.058634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.058572 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data\") pod \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\" (UID: \"c2a81c8f-9198-4d4b-b564-0c288147c8bb\") " Nov 01 09:05:45.059379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.059348 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:45.059564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.059543 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:45.059635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.059588 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:45.059846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.059802 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:45.059964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.059897 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:05:45.062134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.062096 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:05:45.062134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.062126 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:05:45.062617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.062212 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:05:45.063013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.062986 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6" pod "c2a81c8f-9198-4d4b-b564-0c288147c8bb" (UID: "c2a81c8f-9198-4d4b-b564-0c288147c8bb"). InnerVolumeSpecName "kube-api-access-tdfq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160518 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160557 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160569 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdfq6\" (UniqueName: \"kubernetes.io/projected/c2a81c8f-9198-4d4b-b564-0c288147c8bb-kube-api-access-tdfq6\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160580 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160589 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160599 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160608 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160618 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a81c8f-9198-4d4b-b564-0c288147c8bb-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.162836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.160627 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a81c8f-9198-4d4b-b564-0c288147c8bb-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:05:45.265801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.265683 5710 generic.go:356] "Generic (PLEG): container finished" podID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerID="79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203" exitCode=0 Nov 01 09:05:45.265999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.265888 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" event={"ID":"c2a81c8f-9198-4d4b-b564-0c288147c8bb","Type":"ContainerDied","Data":"79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203"} Nov 01 09:05:45.265999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.265932 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" event={"ID":"c2a81c8f-9198-4d4b-b564-0c288147c8bb","Type":"ContainerDied","Data":"cf4cd089663c19dffa64fd58f849baf3e3d8166f437b4b2b09889aaf267f4af9"} Nov 01 09:05:45.265999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.265952 5710 scope.go:122] "RemoveContainer" containerID="79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203" Nov 01 09:05:45.266185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.266097 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl" Nov 01 09:05:45.278316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.278287 5710 scope.go:122] "RemoveContainer" containerID="79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203" Nov 01 09:05:45.278640 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:45.278620 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203\": container with ID starting with 79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203 not found: ID does not exist" containerID="79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203" Nov 01 09:05:45.278742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.278654 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203"} err="failed to get container status \"79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203\": rpc error: code = NotFound desc = could not find container \"79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203\": container with ID starting with 79db9a252f330a22ca78a0251f6b04036137d1c88945a6192feb542c716b3203 not found: ID does not exist" Nov 01 09:05:45.436700 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.436641 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl"] Nov 01 09:05:45.443322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:45.443283 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-l6pnl"] Nov 01 09:05:45.480444 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:05:45.480393 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:05:47.059456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:47.059422 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" path="/var/lib/kubelet/pods/c2a81c8f-9198-4d4b-b564-0c288147c8bb/volumes" Nov 01 09:05:53.520925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:53.520875 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:05:53.715969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:53.520964 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:05:55.388877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.388839 5710 generic.go:356] "Generic (PLEG): container finished" podID="462e1c7c-0c60-4311-a492-de0082e58c32" containerID="036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f" exitCode=0 Nov 01 09:05:55.389475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.388924 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" event={"ID":"462e1c7c-0c60-4311-a492-de0082e58c32","Type":"ContainerDied","Data":"036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f"} Nov 01 09:05:55.389552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.389483 5710 scope.go:122] "RemoveContainer" containerID="036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f" Nov 01 09:05:55.390970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.390946 5710 generic.go:356] "Generic (PLEG): container finished" podID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerID="629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea" exitCode=0 Nov 01 09:05:55.391093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.390989 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" event={"ID":"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3","Type":"ContainerDied","Data":"629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea"} Nov 01 09:05:55.391312 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.391285 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:55.391381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.391342 5710 scope.go:122] "RemoveContainer" containerID="629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea" Nov 01 09:05:55.391441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.391381 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:55.391506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.391433 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:55.392833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.392805 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:55.392917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.392887 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:55.392956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:55.392921 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:05:56.438664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:56.438627 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" event={"ID":"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3","Type":"ContainerStarted","Data":"ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d"} Nov 01 09:05:56.442352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:56.442319 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" event={"ID":"462e1c7c-0c60-4311-a492-de0082e58c32","Type":"ContainerStarted","Data":"adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36"} Nov 01 09:05:56.894300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:56.894259 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:05:56.894768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:56.894643 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 09:05:56.894981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:56.894955 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 09:05:57.361180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:57.361140 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:05:57.362887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:57.362855 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 09:05:57.363034 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:57.362910 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 09:05:57.894678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:57.894159 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 09:05:57.894678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:57.894265 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 09:05:58.361932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:58.361897 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 09:05:58.362201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:58.362178 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 09:05:58.912755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:58.901302 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 09:05:58.934330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:58.934086 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 09:05:59.249888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.249803 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4"] Nov 01 09:05:59.250083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.250052 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" containerID="cri-o://654f420acd8338e9e037af97e05919de2b7eb3fea6818b48c9b1c960ca0373d8" gracePeriod=30 Nov 01 09:05:59.361945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.361909 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 09:05:59.362226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.362199 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 09:05:59.614855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.614814 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:05:59.896411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.896302 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 09:05:59.896747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:05:59.896682 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 09:06:00.361591 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:00.361548 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 09:06:00.362064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:00.361616 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 09:06:00.894273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:00.894225 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 09:06:00.894641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:00.894602 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 09:06:01.361608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:01.361569 5710 patch_prober.go:29] interesting pod/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 09:06:01.362059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:01.361624 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 09:06:01.894260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:01.894218 5710 patch_prober.go:29] interesting pod/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 09:06:01.894480 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:01.894276 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 09:06:02.101528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:02.101163 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v"] Nov 01 09:06:02.366473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:02.366438 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:06:02.367541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:02.367522 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:06:02.900219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:02.900186 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:06:02.900537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:02.900512 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:06:02.901545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:02.901522 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:06:04.161943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.161845 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-292m4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:06:04.161943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.161902 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:06:04.561642 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.561607 5710 generic.go:356] "Generic (PLEG): container finished" podID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerID="654f420acd8338e9e037af97e05919de2b7eb3fea6818b48c9b1c960ca0373d8" exitCode=0 Nov 01 09:06:04.561833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.561674 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" event={"ID":"d2a6db99-8c3d-4882-bf3f-35a6269f9a71","Type":"ContainerDied","Data":"654f420acd8338e9e037af97e05919de2b7eb3fea6818b48c9b1c960ca0373d8"} Nov 01 09:06:04.561833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.561749 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" event={"ID":"d2a6db99-8c3d-4882-bf3f-35a6269f9a71","Type":"ContainerDied","Data":"0c4f9700e52a17e3c0898fe6d878d4e6147f89ebbb15c34c84e93c92084f5723"} Nov 01 09:06:04.561833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.561780 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4f9700e52a17e3c0898fe6d878d4e6147f89ebbb15c34c84e93c92084f5723" Nov 01 09:06:04.574056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.574024 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 09:06:04.578083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578052 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578120 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578286 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578389 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:04.578476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578461 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578544 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578528 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578646 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578615 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578705 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578791 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.578888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578870 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:04.578949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.578878 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert\") pod \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\" (UID: \"d2a6db99-8c3d-4882-bf3f-35a6269f9a71\") " Nov 01 09:06:04.579192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.579171 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:04.579967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.579941 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.580083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.579970 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.580083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.579997 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.580931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.580879 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:04.581063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.580923 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:04.582212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.582188 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "kube-api-access-jpgx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:04.582300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.582234 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:06:04.582443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.582424 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:04.583876 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.583854 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token" pod "d2a6db99-8c3d-4882-bf3f-35a6269f9a71" (UID: "d2a6db99-8c3d-4882-bf3f-35a6269f9a71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:04.681201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.681163 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.681201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.681195 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpgx7\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-kube-api-access-jpgx7\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.681201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.681206 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.681201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.681214 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.681492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.681229 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:04.681492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:04.681239 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a6db99-8c3d-4882-bf3f-35a6269f9a71-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:05.569445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:05.569410 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4" Nov 01 09:06:05.771057 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:05.771017 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4"] Nov 01 09:06:05.774933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:05.774899 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-292m4"] Nov 01 09:06:07.060162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.060124 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" path="/var/lib/kubelet/pods/d2a6db99-8c3d-4882-bf3f-35a6269f9a71/volumes" Nov 01 09:06:07.102607 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.102555 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" podUID="86dfbc06-96bb-4fe4-881c-110ffe5095e5" containerName="payload-processing" containerID="cri-o://acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632" gracePeriod=25 Nov 01 09:06:07.393362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.393331 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:06:07.418745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.418687 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst\" (UniqueName: \"kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst\") pod \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " Nov 01 09:06:07.418943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.418843 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume\") pod \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\" (UID: \"86dfbc06-96bb-4fe4-881c-110ffe5095e5\") " Nov 01 09:06:07.419235 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.419195 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume" pod "86dfbc06-96bb-4fe4-881c-110ffe5095e5" (UID: "86dfbc06-96bb-4fe4-881c-110ffe5095e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:07.419359 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.419322 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86dfbc06-96bb-4fe4-881c-110ffe5095e5-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:07.421171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.421149 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst" pod "86dfbc06-96bb-4fe4-881c-110ffe5095e5" (UID: "86dfbc06-96bb-4fe4-881c-110ffe5095e5"). InnerVolumeSpecName "kube-api-access-wqjst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:07.520535 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.520504 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqjst\" (UniqueName: \"kubernetes.io/projected/86dfbc06-96bb-4fe4-881c-110ffe5095e5-kube-api-access-wqjst\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:07.585903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.585866 5710 generic.go:356] "Generic (PLEG): container finished" podID="86dfbc06-96bb-4fe4-881c-110ffe5095e5" containerID="acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632" exitCode=0 Nov 01 09:06:07.586119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.585918 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" event={"ID":"86dfbc06-96bb-4fe4-881c-110ffe5095e5","Type":"ContainerDied","Data":"acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632"} Nov 01 09:06:07.586119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.585937 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" Nov 01 09:06:07.586119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.585990 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v" event={"ID":"86dfbc06-96bb-4fe4-881c-110ffe5095e5","Type":"ContainerDied","Data":"674eff148e636af59d4555a7328a2ad4e7f746ee21e1f211e4b44f8ae402ac55"} Nov 01 09:06:07.586119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.586024 5710 scope.go:122] "RemoveContainer" containerID="acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632" Nov 01 09:06:07.600224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.600199 5710 scope.go:122] "RemoveContainer" containerID="acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632" Nov 01 09:06:07.600601 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:07.600582 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632\": container with ID starting with acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632 not found: ID does not exist" containerID="acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632" Nov 01 09:06:07.600732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.600607 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632"} err="failed to get container status \"acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632\": rpc error: code = NotFound desc = could not find container \"acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632\": container with ID starting with acc052ea5470ac749902ee7d25dd32670de02d19c9776e86926ccf2d3ed96632 not found: ID does not exist" Nov 01 09:06:07.740990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.740899 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v"] Nov 01 09:06:07.747565 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:07.747534 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-64dc5ccd9-g5r7v"] Nov 01 09:06:08.927324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:08.927286 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf"] Nov 01 09:06:08.927848 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:08.927530 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" podUID="b201ed59-c659-494f-902c-848b18ee8c68" containerName="maas-api" containerID="cri-o://934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069" gracePeriod=30 Nov 01 09:06:09.059930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.059898 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86dfbc06-96bb-4fe4-881c-110ffe5095e5" path="/var/lib/kubelet/pods/86dfbc06-96bb-4fe4-881c-110ffe5095e5/volumes" Nov 01 09:06:09.262929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.262894 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:06:09.344744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.344269 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs\") pod \"b201ed59-c659-494f-902c-848b18ee8c68\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " Nov 01 09:06:09.344744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.344438 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls\") pod \"b201ed59-c659-494f-902c-848b18ee8c68\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " Nov 01 09:06:09.344744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.344533 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g\" (UniqueName: \"kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g\") pod \"b201ed59-c659-494f-902c-848b18ee8c68\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " Nov 01 09:06:09.344744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.344608 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca\") pod \"b201ed59-c659-494f-902c-848b18ee8c68\" (UID: \"b201ed59-c659-494f-902c-848b18ee8c68\") " Nov 01 09:06:09.352799 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.345450 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca" pod "b201ed59-c659-494f-902c-848b18ee8c68" (UID: "b201ed59-c659-494f-902c-848b18ee8c68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:09.379964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.379901 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls" pod "b201ed59-c659-494f-902c-848b18ee8c68" (UID: "b201ed59-c659-494f-902c-848b18ee8c68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:06:09.380162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.380097 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs" pod "b201ed59-c659-494f-902c-848b18ee8c68" (UID: "b201ed59-c659-494f-902c-848b18ee8c68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:06:09.380337 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.380302 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g" pod "b201ed59-c659-494f-902c-848b18ee8c68" (UID: "b201ed59-c659-494f-902c-848b18ee8c68"). InnerVolumeSpecName "kube-api-access-2md5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:09.388217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.388181 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt"] Nov 01 09:06:09.388437 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.388413 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" podUID="29ac004d-6065-4aeb-a587-344df49951b1" containerName="payload-processing" containerID="cri-o://205fb41e59d6b34c75f6225799435717609cdb35265dcd3f6cb63092b6b01d8e" gracePeriod=30 Nov 01 09:06:09.446574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.446541 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd"] Nov 01 09:06:09.446998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.446967 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" podUID="1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" containerName="payload-pre-processing" containerID="cri-o://d3d6ad1cd9540a11749522794c4fa9b4dedd1d11e752b4321628fa6b8fdf2323" gracePeriod=30 Nov 01 09:06:09.447140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.446581 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2md5g\" (UniqueName: \"kubernetes.io/projected/b201ed59-c659-494f-902c-848b18ee8c68-kube-api-access-2md5g\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.447237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.447157 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b201ed59-c659-494f-902c-848b18ee8c68-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.447237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.447175 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.447237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.447189 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b201ed59-c659-494f-902c-848b18ee8c68-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.605111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.605071 5710 generic.go:356] "Generic (PLEG): container finished" podID="b201ed59-c659-494f-902c-848b18ee8c68" containerID="934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069" exitCode=0 Nov 01 09:06:09.605310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.605142 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" Nov 01 09:06:09.605310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.605187 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" event={"ID":"b201ed59-c659-494f-902c-848b18ee8c68","Type":"ContainerDied","Data":"934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069"} Nov 01 09:06:09.605310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.605229 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf" event={"ID":"b201ed59-c659-494f-902c-848b18ee8c68","Type":"ContainerDied","Data":"7102e45756436a865c6cb258aa41cb82bebc731b4b6b7fa46ee83f616785a8f0"} Nov 01 09:06:09.605310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.605250 5710 scope.go:122] "RemoveContainer" containerID="934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069" Nov 01 09:06:09.612330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.611946 5710 generic.go:356] "Generic (PLEG): container finished" podID="29ac004d-6065-4aeb-a587-344df49951b1" containerID="205fb41e59d6b34c75f6225799435717609cdb35265dcd3f6cb63092b6b01d8e" exitCode=0 Nov 01 09:06:09.612330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.612155 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" event={"ID":"29ac004d-6065-4aeb-a587-344df49951b1","Type":"ContainerDied","Data":"205fb41e59d6b34c75f6225799435717609cdb35265dcd3f6cb63092b6b01d8e"} Nov 01 09:06:09.614417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.614301 5710 generic.go:356] "Generic (PLEG): container finished" podID="1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" containerID="d3d6ad1cd9540a11749522794c4fa9b4dedd1d11e752b4321628fa6b8fdf2323" exitCode=0 Nov 01 09:06:09.614417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.614370 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" event={"ID":"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2","Type":"ContainerDied","Data":"d3d6ad1cd9540a11749522794c4fa9b4dedd1d11e752b4321628fa6b8fdf2323"} Nov 01 09:06:09.620342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.620311 5710 scope.go:122] "RemoveContainer" containerID="934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069" Nov 01 09:06:09.620774 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:09.620699 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069\": container with ID starting with 934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069 not found: ID does not exist" containerID="934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069" Nov 01 09:06:09.620888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.620786 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069"} err="failed to get container status \"934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069\": rpc error: code = NotFound desc = could not find container \"934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069\": container with ID starting with 934d08adfa426d51a481127a4fa89d163ec11956b12a15792253f016afddb069 not found: ID does not exist" Nov 01 09:06:09.710253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.710222 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:06:09.753055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.753024 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:06:09.755237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.755199 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh\" (UniqueName: \"kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh\") pod \"29ac004d-6065-4aeb-a587-344df49951b1\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " Nov 01 09:06:09.765019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.755855 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume\") pod \"29ac004d-6065-4aeb-a587-344df49951b1\" (UID: \"29ac004d-6065-4aeb-a587-344df49951b1\") " Nov 01 09:06:09.765019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.756347 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume" pod "29ac004d-6065-4aeb-a587-344df49951b1" (UID: "29ac004d-6065-4aeb-a587-344df49951b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:09.765019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.757313 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ac004d-6065-4aeb-a587-344df49951b1-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.765019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.760381 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh" pod "29ac004d-6065-4aeb-a587-344df49951b1" (UID: "29ac004d-6065-4aeb-a587-344df49951b1"). InnerVolumeSpecName "kube-api-access-zgtrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:09.771582 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.771544 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf"] Nov 01 09:06:09.775697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.775659 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-991c28-76cd46d65b-flvcf"] Nov 01 09:06:09.858788 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.858752 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume\") pod \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " Nov 01 09:06:09.858952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.858819 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f\" (UniqueName: \"kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f\") pod \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\" (UID: \"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2\") " Nov 01 09:06:09.859161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.859137 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume" pod "1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" (UID: "1ae79c83-4f36-45ec-af2f-d792fdd9e9e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:09.859307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.859292 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgtrh\" (UniqueName: \"kubernetes.io/projected/29ac004d-6065-4aeb-a587-344df49951b1-kube-api-access-zgtrh\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.859376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.859310 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:09.861153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:09.861125 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f" pod "1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" (UID: "1ae79c83-4f36-45ec-af2f-d792fdd9e9e2"). InnerVolumeSpecName "kube-api-access-d789f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:10.003484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.001777 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d789f\" (UniqueName: \"kubernetes.io/projected/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2-kube-api-access-d789f\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.013828 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg"] Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014800 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b201ed59-c659-494f-902c-848b18ee8c68" containerName="maas-api" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014822 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b201ed59-c659-494f-902c-848b18ee8c68" containerName="maas-api" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014838 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" containerName="payload-pre-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014847 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" containerName="payload-pre-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014856 5710 cpu_manager.go:414] "containerMap: removing container" podUID="29ac004d-6065-4aeb-a587-344df49951b1" containerName="payload-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014864 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ac004d-6065-4aeb-a587-344df49951b1" containerName="payload-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014880 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014887 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014913 5710 cpu_manager.go:414] "containerMap: removing container" podUID="86dfbc06-96bb-4fe4-881c-110ffe5095e5" containerName="payload-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014920 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86dfbc06-96bb-4fe4-881c-110ffe5095e5" containerName="payload-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014964 5710 cpu_manager.go:414] "containerMap: removing container" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014972 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014981 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.014989 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015136 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b71a780-fc73-4bcf-979e-16bcd0ba5765" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015151 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a6db99-8c3d-4882-bf3f-35a6269f9a71" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015166 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b201ed59-c659-494f-902c-848b18ee8c68" containerName="maas-api" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015175 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" containerName="payload-pre-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015183 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ac004d-6065-4aeb-a587-344df49951b1" containerName="payload-processing" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015192 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a81c8f-9198-4d4b-b564-0c288147c8bb" containerName="istio-proxy" Nov 01 09:06:10.020815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.015203 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="86dfbc06-96bb-4fe4-881c-110ffe5095e5" containerName="payload-processing" Nov 01 09:06:10.022441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.022410 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.043145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.043109 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg"] Nov 01 09:06:10.073463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.073424 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0460d\"" Nov 01 09:06:10.103345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.103294 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhnz\" (UniqueName: \"kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.103777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.103745 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.201802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.199589 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk"] Nov 01 09:06:10.212746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.208672 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.212746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.209783 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhnz\" (UniqueName: \"kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.212746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.209928 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.224745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.216968 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.234795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.232109 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk"] Nov 01 09:06:10.259445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.259409 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0460d-dockercfg-vptf4\"" Nov 01 09:06:10.275585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.275529 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhnz\" (UniqueName: \"kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz\") pod \"payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.310792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.310739 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.311038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.310807 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzc2\" (UniqueName: \"kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2\") pod \"payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.353414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.353362 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:10.412814 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.412775 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.412984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.412841 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzc2\" (UniqueName: \"kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2\") pod \"payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.414045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.414013 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.471184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.471114 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzc2\" (UniqueName: \"kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2\") pod \"payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.549828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.549789 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:10.642286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.642243 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" event={"ID":"29ac004d-6065-4aeb-a587-344df49951b1","Type":"ContainerDied","Data":"59cdeb0ccf640434709d1102528da4a344be4829252123ee3ab7563188b490cb"} Nov 01 09:06:10.642479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.642324 5710 scope.go:122] "RemoveContainer" containerID="205fb41e59d6b34c75f6225799435717609cdb35265dcd3f6cb63092b6b01d8e" Nov 01 09:06:10.642541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.642482 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt" Nov 01 09:06:10.648268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.648240 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" event={"ID":"1ae79c83-4f36-45ec-af2f-d792fdd9e9e2","Type":"ContainerDied","Data":"3c17b3c106eb8f2662c77e26638101d5fd6951120ffc5e88ecf5670d18066946"} Nov 01 09:06:10.648426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.648352 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd" Nov 01 09:06:10.659851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.659825 5710 scope.go:122] "RemoveContainer" containerID="d3d6ad1cd9540a11749522794c4fa9b4dedd1d11e752b4321628fa6b8fdf2323" Nov 01 09:06:10.743163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.743080 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg"] Nov 01 09:06:10.749911 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:06:10.749882 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe3ec1_9c61_412e_b1d1_149552dd46b5.slice/crio-64801bdf06e33a8b1d6882c6a303c8ce0705f3feb9a12db2cb78127d9cab7fac WatchSource:0}: Error finding container 64801bdf06e33a8b1d6882c6a303c8ce0705f3feb9a12db2cb78127d9cab7fac: Status 404 returned error can't find the container with id 64801bdf06e33a8b1d6882c6a303c8ce0705f3feb9a12db2cb78127d9cab7fac Nov 01 09:06:10.801178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.801148 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd"] Nov 01 09:06:10.805590 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.805561 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-991c28-78fb777dd7-4xjmd"] Nov 01 09:06:10.909664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.909631 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt"] Nov 01 09:06:10.915552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.915516 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-991c28-b89cf5ddb-4xnwt"] Nov 01 09:06:10.933281 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:10.933245 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk"] Nov 01 09:06:10.940237 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:06:10.940203 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220dd668_434b_445a_92ab_a9846e908061.slice/crio-950b56071dd7cbc25944b9e5b0a4b3de941c1aa1e2159df9d08259f2b95bf196 WatchSource:0}: Error finding container 950b56071dd7cbc25944b9e5b0a4b3de941c1aa1e2159df9d08259f2b95bf196: Status 404 returned error can't find the container with id 950b56071dd7cbc25944b9e5b0a4b3de941c1aa1e2159df9d08259f2b95bf196 Nov 01 09:06:11.060886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.060856 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae79c83-4f36-45ec-af2f-d792fdd9e9e2" path="/var/lib/kubelet/pods/1ae79c83-4f36-45ec-af2f-d792fdd9e9e2/volumes" Nov 01 09:06:11.061365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.061350 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ac004d-6065-4aeb-a587-344df49951b1" path="/var/lib/kubelet/pods/29ac004d-6065-4aeb-a587-344df49951b1/volumes" Nov 01 09:06:11.061819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.061802 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b201ed59-c659-494f-902c-848b18ee8c68" path="/var/lib/kubelet/pods/b201ed59-c659-494f-902c-848b18ee8c68/volumes" Nov 01 09:06:11.658074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.658033 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" event={"ID":"6abe3ec1-9c61-412e-b1d1-149552dd46b5","Type":"ContainerStarted","Data":"145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158"} Nov 01 09:06:11.658255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.658086 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" event={"ID":"6abe3ec1-9c61-412e-b1d1-149552dd46b5","Type":"ContainerStarted","Data":"64801bdf06e33a8b1d6882c6a303c8ce0705f3feb9a12db2cb78127d9cab7fac"} Nov 01 09:06:11.658255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.658233 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:11.659918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.659891 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" event={"ID":"220dd668-434b-445a-92ab-a9846e908061","Type":"ContainerStarted","Data":"2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc"} Nov 01 09:06:11.660089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.659930 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" event={"ID":"220dd668-434b-445a-92ab-a9846e908061","Type":"ContainerStarted","Data":"950b56071dd7cbc25944b9e5b0a4b3de941c1aa1e2159df9d08259f2b95bf196"} Nov 01 09:06:11.660089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.659980 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:11.780046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.779985 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" podStartSLOduration=2.7799690139999997 podStartE2EDuration="2.779969014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:11.777706336 +0000 UTC m=+1994.866043011" watchObservedRunningTime="2026-11-01 09:06:11.779969014 +0000 UTC m=+1994.868305677" Nov 01 09:06:11.898744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:11.894987 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" podStartSLOduration=1.894776914 podStartE2EDuration="1.894776914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:11.887115011 +0000 UTC m=+1994.975451891" watchObservedRunningTime="2026-11-01 09:06:11.894776914 +0000 UTC m=+1994.983113630" Nov 01 09:06:15.479828 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:15.479787 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:06:19.326483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:19.326438 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp"] Nov 01 09:06:19.326964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:19.326676 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" containerID="cri-o://cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f" gracePeriod=30 Nov 01 09:06:22.674952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:22.674916 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:06:22.675530 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:22.675003 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:06:24.596560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.596532 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:06:24.629033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.628999 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629131 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629171 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629209 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629259 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629292 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:24.629484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629380 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629416 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629486 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket\") pod \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\" (UID: \"bac2d1a7-18ca-4294-b82c-4607fcad47d0\") " Nov 01 09:06:24.629628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629560 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:24.629869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.629838 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:24.630263 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.630243 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.630393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.630267 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.630393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.630305 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.630393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.630232 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:24.630581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.630562 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:24.631873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.631850 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:06:24.632138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.632108 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:24.632742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.632692 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "kube-api-access-zvzz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:24.633332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.633307 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo" pod "bac2d1a7-18ca-4294-b82c-4607fcad47d0" (UID: "bac2d1a7-18ca-4294-b82c-4607fcad47d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:06:24.731453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.731418 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.731453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.731447 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.731453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.731460 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.731763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.731473 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac2d1a7-18ca-4294-b82c-4607fcad47d0-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.731763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.731482 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvzz9\" (UniqueName: \"kubernetes.io/projected/bac2d1a7-18ca-4294-b82c-4607fcad47d0-kube-api-access-zvzz9\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.731763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.731494 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac2d1a7-18ca-4294-b82c-4607fcad47d0-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:24.761691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.761641 5710 generic.go:356] "Generic (PLEG): container finished" podID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerID="cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f" exitCode=0 Nov 01 09:06:24.761914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.761737 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" Nov 01 09:06:24.761914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.761861 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" event={"ID":"bac2d1a7-18ca-4294-b82c-4607fcad47d0","Type":"ContainerDied","Data":"cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f"} Nov 01 09:06:24.762028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.761926 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp" event={"ID":"bac2d1a7-18ca-4294-b82c-4607fcad47d0","Type":"ContainerDied","Data":"e205b4ea3246a29a4c3b16d76e5eabb2a59111fc84f9661a755cba2d40e91b88"} Nov 01 09:06:24.762028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.761958 5710 scope.go:122] "RemoveContainer" containerID="cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f" Nov 01 09:06:24.783067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.776274 5710 scope.go:122] "RemoveContainer" containerID="1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4" Nov 01 09:06:24.788548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.788523 5710 scope.go:122] "RemoveContainer" containerID="cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f" Nov 01 09:06:24.789203 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:24.789177 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f\": container with ID starting with cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f not found: ID does not exist" containerID="cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f" Nov 01 09:06:24.789335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.789209 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f"} err="failed to get container status \"cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f\": rpc error: code = NotFound desc = could not find container \"cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f\": container with ID starting with cc7c2095eed5c5164810d1e668a133ec29e02b6ecc06ff4c507eae988a99d17f not found: ID does not exist" Nov 01 09:06:24.789335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.789229 5710 scope.go:122] "RemoveContainer" containerID="1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4" Nov 01 09:06:24.789535 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:24.789515 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4\": container with ID starting with 1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4 not found: ID does not exist" containerID="1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4" Nov 01 09:06:24.789613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.789538 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4"} err="failed to get container status \"1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4\": rpc error: code = NotFound desc = could not find container \"1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4\": container with ID starting with 1fbb8429c68ef04bce98d83e5c98155bfa495efff09e5d17a829e3c8048e1fb4 not found: ID does not exist" Nov 01 09:06:24.918614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.918512 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp"] Nov 01 09:06:24.923242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:24.923168 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-991c28-openshift-default-59ccc767bc-2tsrp"] Nov 01 09:06:25.059440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:25.059401 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" path="/var/lib/kubelet/pods/bac2d1a7-18ca-4294-b82c-4607fcad47d0/volumes" Nov 01 09:06:25.088562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:25.088521 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:06:31.106745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.101003 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4"] Nov 01 09:06:31.969094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969054 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5"] Nov 01 09:06:31.969805 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969786 5710 cpu_manager.go:414] "containerMap: removing container" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" Nov 01 09:06:31.969805 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969806 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" Nov 01 09:06:31.969972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969832 5710 cpu_manager.go:414] "containerMap: removing container" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" Nov 01 09:06:31.969972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969839 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" Nov 01 09:06:31.969972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969932 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" Nov 01 09:06:31.969972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.969943 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2d1a7-18ca-4294-b82c-4607fcad47d0" containerName="istio-proxy" Nov 01 09:06:31.975225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:31.975187 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.026236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.026201 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-23fa20\"" Nov 01 09:06:32.026483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.026410 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-23fa20-dockercfg-kgwx9\"" Nov 01 09:06:32.029089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.028931 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5"] Nov 01 09:06:32.064408 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.064364 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.064635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.064432 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjh2h\" (UniqueName: \"kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.165809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.165761 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.166472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.165825 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjh2h\" (UniqueName: \"kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.166909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.166884 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.226682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.226601 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjh2h\" (UniqueName: \"kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h\") pod \"payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.291144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.291109 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:32.742599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.742036 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5"] Nov 01 09:06:32.761044 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:06:32.761007 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33799e41_a6e0_431e_ba21_ae3ccd50c8ba.slice/crio-2ef077c3687d3ab4de6fdce27035ffc9fef944eb2f8804aada51e06099e625bb WatchSource:0}: Error finding container 2ef077c3687d3ab4de6fdce27035ffc9fef944eb2f8804aada51e06099e625bb: Status 404 returned error can't find the container with id 2ef077c3687d3ab4de6fdce27035ffc9fef944eb2f8804aada51e06099e625bb Nov 01 09:06:32.819280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:32.819242 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" event={"ID":"33799e41-a6e0-431e-ba21-ae3ccd50c8ba","Type":"ContainerStarted","Data":"2ef077c3687d3ab4de6fdce27035ffc9fef944eb2f8804aada51e06099e625bb"} Nov 01 09:06:33.827795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:33.827746 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" event={"ID":"33799e41-a6e0-431e-ba21-ae3ccd50c8ba","Type":"ContainerStarted","Data":"77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951"} Nov 01 09:06:33.828288 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:33.827833 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:33.937608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:33.937555 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" podStartSLOduration=2.937539709 podStartE2EDuration="2.937539709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:33.936969936 +0000 UTC m=+2017.025306605" watchObservedRunningTime="2026-11-01 09:06:33.937539709 +0000 UTC m=+2017.025876380" Nov 01 09:06:36.102214 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.102147 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" podUID="58ef9834-8cd3-448b-a979-7bcb4481d4d7" containerName="payload-processing" containerID="cri-o://e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37" gracePeriod=25 Nov 01 09:06:36.376575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.376538 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:06:36.423598 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.423557 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6\" (UniqueName: \"kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6\") pod \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " Nov 01 09:06:36.423836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.423642 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume\") pod \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\" (UID: \"58ef9834-8cd3-448b-a979-7bcb4481d4d7\") " Nov 01 09:06:36.424050 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.424027 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume" pod "58ef9834-8cd3-448b-a979-7bcb4481d4d7" (UID: "58ef9834-8cd3-448b-a979-7bcb4481d4d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:36.424188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.424176 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ef9834-8cd3-448b-a979-7bcb4481d4d7-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:36.426185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.426157 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6" pod "58ef9834-8cd3-448b-a979-7bcb4481d4d7" (UID: "58ef9834-8cd3-448b-a979-7bcb4481d4d7"). InnerVolumeSpecName "kube-api-access-99jn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:36.525212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.525162 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jn6\" (UniqueName: \"kubernetes.io/projected/58ef9834-8cd3-448b-a979-7bcb4481d4d7-kube-api-access-99jn6\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:36.852574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.852535 5710 generic.go:356] "Generic (PLEG): container finished" podID="58ef9834-8cd3-448b-a979-7bcb4481d4d7" containerID="e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37" exitCode=0 Nov 01 09:06:36.852834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.852586 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" Nov 01 09:06:36.852834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.852614 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" event={"ID":"58ef9834-8cd3-448b-a979-7bcb4481d4d7","Type":"ContainerDied","Data":"e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37"} Nov 01 09:06:36.852834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.852671 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4" event={"ID":"58ef9834-8cd3-448b-a979-7bcb4481d4d7","Type":"ContainerDied","Data":"03161284dea6caaf3e8053b3496c2a9480615d93f425c9082538a46bad7fe83c"} Nov 01 09:06:36.852834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.852692 5710 scope.go:122] "RemoveContainer" containerID="e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37" Nov 01 09:06:36.864562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.864538 5710 scope.go:122] "RemoveContainer" containerID="e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37" Nov 01 09:06:36.864997 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:36.864970 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37\": container with ID starting with e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37 not found: ID does not exist" containerID="e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37" Nov 01 09:06:36.865099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:36.865008 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37"} err="failed to get container status \"e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37\": rpc error: code = NotFound desc = could not find container \"e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37\": container with ID starting with e260c3c970a95f9c83200f200bdd2546e36582bc150fac171b9ecb5a3f2bfc37 not found: ID does not exist" Nov 01 09:06:37.008638 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:37.008603 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4"] Nov 01 09:06:37.020767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:37.020732 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb28ec-77967c87f7-xczq4"] Nov 01 09:06:37.059324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:37.059280 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ef9834-8cd3-448b-a979-7bcb4481d4d7" path="/var/lib/kubelet/pods/58ef9834-8cd3-448b-a979-7bcb4481d4d7/volumes" Nov 01 09:06:38.497782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.497743 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg"] Nov 01 09:06:38.498471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.498448 5710 cpu_manager.go:414] "containerMap: removing container" podUID="58ef9834-8cd3-448b-a979-7bcb4481d4d7" containerName="payload-processing" Nov 01 09:06:38.498586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.498476 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ef9834-8cd3-448b-a979-7bcb4481d4d7" containerName="payload-processing" Nov 01 09:06:38.498736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.498704 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ef9834-8cd3-448b-a979-7bcb4481d4d7" containerName="payload-processing" Nov 01 09:06:38.504089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.504061 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.549695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.549652 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.549949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.549817 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.549949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.549846 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.550148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.550111 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x49cz\" (UniqueName: \"kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.600654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.600615 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-eaabf4-serving-cert\"" Nov 01 09:06:38.603489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.603462 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg"] Nov 01 09:06:38.651972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.651924 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x49cz\" (UniqueName: \"kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.652163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.652068 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.652163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.652135 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.652282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.652158 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.652342 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:38.652282 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-serving-cert: secret "maas-api-e2e-ipp-a-eaabf4-serving-cert" not found Nov 01 09:06:38.652387 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:38.652353 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls podName:517fc65a-4127-4b76-946b-2b46862047d4 nodeName:}" failed. No retries permitted until 2026-11-01 09:06:39.152331598 +0000 UTC m=+2022.240668266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls") pod "maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" (UID: "517fc65a-4127-4b76-946b-2b46862047d4") : secret "maas-api-e2e-ipp-a-eaabf4-serving-cert" not found Nov 01 09:06:38.652984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.652910 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.654822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.654783 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:38.710843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:38.710807 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x49cz\" (UniqueName: \"kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:39.160603 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:39.160550 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:39.163332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:39.163307 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:39.430003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:39.429897 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:39.817165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:39.817071 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg"] Nov 01 09:06:39.823981 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:06:39.823946 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517fc65a_4127_4b76_946b_2b46862047d4.slice/crio-0e56d622a0e42cb1d11185c28ad8fe0bd02c721068a1a6d9870197c33284a51a WatchSource:0}: Error finding container 0e56d622a0e42cb1d11185c28ad8fe0bd02c721068a1a6d9870197c33284a51a: Status 404 returned error can't find the container with id 0e56d622a0e42cb1d11185c28ad8fe0bd02c721068a1a6d9870197c33284a51a Nov 01 09:06:39.825470 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:39.825449 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:06:39.873239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:39.873204 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" event={"ID":"517fc65a-4127-4b76-946b-2b46862047d4","Type":"ContainerStarted","Data":"0e56d622a0e42cb1d11185c28ad8fe0bd02c721068a1a6d9870197c33284a51a"} Nov 01 09:06:41.897919 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:41.897879 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" event={"ID":"517fc65a-4127-4b76-946b-2b46862047d4","Type":"ContainerStarted","Data":"2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210"} Nov 01 09:06:41.898360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:41.897944 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:42.018341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:42.018282 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" podStartSLOduration=2.904517244 podStartE2EDuration="4.018262886s" totalImagesPullingTime="1.113745642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:42.015919923 +0000 UTC m=+2025.104256602" watchObservedRunningTime="2026-11-01 09:06:42.018262886 +0000 UTC m=+2025.106599558" Nov 01 09:06:42.651633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:42.651593 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf"] Nov 01 09:06:42.753019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:42.752976 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64"] Nov 01 09:06:44.839451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:44.839419 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:06:45.480731 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:45.480668 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:06:47.652126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.652083 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" podUID="1f2e28bb-3769-4a72-9463-7cb1b95667c6" containerName="payload-processing" containerID="cri-o://26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027" gracePeriod=25 Nov 01 09:06:47.753819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.753774 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" podUID="cda72c59-82cd-48a0-a17b-e75a0289a8ad" containerName="payload-pre-processing" containerID="cri-o://9e559c54b2d4864e6f43cc507b0d6a254dd18cdfae2834d5cac08439179e0271" gracePeriod=25 Nov 01 09:06:47.910663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.910633 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.948118 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.951758 5710 generic.go:356] "Generic (PLEG): container finished" podID="1f2e28bb-3769-4a72-9463-7cb1b95667c6" containerID="26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027" exitCode=0 Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.951808 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.951840 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" event={"ID":"1f2e28bb-3769-4a72-9463-7cb1b95667c6","Type":"ContainerDied","Data":"26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027"} Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.951901 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf" event={"ID":"1f2e28bb-3769-4a72-9463-7cb1b95667c6","Type":"ContainerDied","Data":"330d20511c0e4cd2f118f03d0aab0d06e3fbe56b41529babd1ce6c3b027f5844"} Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.951932 5710 scope.go:122] "RemoveContainer" containerID="26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027" Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.954303 5710 generic.go:356] "Generic (PLEG): container finished" podID="cda72c59-82cd-48a0-a17b-e75a0289a8ad" containerID="9e559c54b2d4864e6f43cc507b0d6a254dd18cdfae2834d5cac08439179e0271" exitCode=0 Nov 01 09:06:47.954886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.954463 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" event={"ID":"cda72c59-82cd-48a0-a17b-e75a0289a8ad","Type":"ContainerDied","Data":"9e559c54b2d4864e6f43cc507b0d6a254dd18cdfae2834d5cac08439179e0271"} Nov 01 09:06:47.965970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.965940 5710 scope.go:122] "RemoveContainer" containerID="26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027" Nov 01 09:06:47.966367 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:06:47.966328 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027\": container with ID starting with 26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027 not found: ID does not exist" containerID="26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027" Nov 01 09:06:47.966473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:47.966363 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027"} err="failed to get container status \"26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027\": rpc error: code = NotFound desc = could not find container \"26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027\": container with ID starting with 26f718beb08fc7fa91e8137badf997c578eb0c3d8b32004451b4042a3586a027 not found: ID does not exist" Nov 01 09:06:48.031991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.031963 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:06:48.088066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.088019 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86\" (UniqueName: \"kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86\") pod \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " Nov 01 09:06:48.088283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.088100 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume\") pod \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\" (UID: \"cda72c59-82cd-48a0-a17b-e75a0289a8ad\") " Nov 01 09:06:48.088283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.088140 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume\") pod \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " Nov 01 09:06:48.088283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.088259 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd\" (UniqueName: \"kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd\") pod \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\" (UID: \"1f2e28bb-3769-4a72-9463-7cb1b95667c6\") " Nov 01 09:06:48.089093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.089055 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume" pod "cda72c59-82cd-48a0-a17b-e75a0289a8ad" (UID: "cda72c59-82cd-48a0-a17b-e75a0289a8ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:48.089256 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.089117 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume" pod "1f2e28bb-3769-4a72-9463-7cb1b95667c6" (UID: "1f2e28bb-3769-4a72-9463-7cb1b95667c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:06:48.091782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.091756 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86" pod "cda72c59-82cd-48a0-a17b-e75a0289a8ad" (UID: "cda72c59-82cd-48a0-a17b-e75a0289a8ad"). InnerVolumeSpecName "kube-api-access-qhg86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:48.091905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.091837 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd" pod "1f2e28bb-3769-4a72-9463-7cb1b95667c6" (UID: "1f2e28bb-3769-4a72-9463-7cb1b95667c6"). InnerVolumeSpecName "kube-api-access-pmfzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:06:48.190552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.190438 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhg86\" (UniqueName: \"kubernetes.io/projected/cda72c59-82cd-48a0-a17b-e75a0289a8ad-kube-api-access-qhg86\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:48.190552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.190473 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda72c59-82cd-48a0-a17b-e75a0289a8ad-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:48.190552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.190483 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2e28bb-3769-4a72-9463-7cb1b95667c6-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:48.190552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.190493 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfzd\" (UniqueName: \"kubernetes.io/projected/1f2e28bb-3769-4a72-9463-7cb1b95667c6-kube-api-access-pmfzd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:06:48.442749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.442612 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf"] Nov 01 09:06:48.446060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.446029 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-5567bb9d8c-n9lvf"] Nov 01 09:06:48.962471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.962444 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" Nov 01 09:06:48.962471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.962458 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64" event={"ID":"cda72c59-82cd-48a0-a17b-e75a0289a8ad","Type":"ContainerDied","Data":"dbe2c4f37a9d02153c96599639e015109d95822ca32263da0838451d631878a4"} Nov 01 09:06:48.963074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:48.962548 5710 scope.go:122] "RemoveContainer" containerID="9e559c54b2d4864e6f43cc507b0d6a254dd18cdfae2834d5cac08439179e0271" Nov 01 09:06:49.081877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.081839 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2e28bb-3769-4a72-9463-7cb1b95667c6" path="/var/lib/kubelet/pods/1f2e28bb-3769-4a72-9463-7cb1b95667c6/volumes" Nov 01 09:06:49.137625 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.137593 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64"] Nov 01 09:06:49.142352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.142320 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-b5c6c4c6c-zrh64"] Nov 01 09:06:49.417432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.417393 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r"] Nov 01 09:06:49.418036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.418022 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1f2e28bb-3769-4a72-9463-7cb1b95667c6" containerName="payload-processing" Nov 01 09:06:49.418036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.418037 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2e28bb-3769-4a72-9463-7cb1b95667c6" containerName="payload-processing" Nov 01 09:06:49.418131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.418049 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cda72c59-82cd-48a0-a17b-e75a0289a8ad" containerName="payload-pre-processing" Nov 01 09:06:49.418131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.418054 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda72c59-82cd-48a0-a17b-e75a0289a8ad" containerName="payload-pre-processing" Nov 01 09:06:49.418198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.418141 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2e28bb-3769-4a72-9463-7cb1b95667c6" containerName="payload-processing" Nov 01 09:06:49.418198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.418150 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda72c59-82cd-48a0-a17b-e75a0289a8ad" containerName="payload-pre-processing" Nov 01 09:06:49.423092 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.423066 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.476746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.474406 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4be7cc-dockercfg-f6vg6\"" Nov 01 09:06:49.476746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.474641 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4be7cc\"" Nov 01 09:06:49.521438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.516344 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.521438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.516494 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5958h\" (UniqueName: \"kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.562451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.562406 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r"] Nov 01 09:06:49.582074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.582004 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr"] Nov 01 09:06:49.587386 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.587356 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.618142 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.618108 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5958h\" (UniqueName: \"kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.618362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.618186 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6jq\" (UniqueName: \"kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq\") pod \"payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.618362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.618250 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.618479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.618355 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.619045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.619027 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.635819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.635778 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr"] Nov 01 09:06:49.676734 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.676645 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5958h\" (UniqueName: \"kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h\") pod \"payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.720184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.720133 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6jq\" (UniqueName: \"kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq\") pod \"payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.720401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.720226 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.720919 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.720899 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.737815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.737785 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:49.782983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.782952 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6jq\" (UniqueName: \"kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq\") pod \"payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:49.901904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:49.901864 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:50.129301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.129247 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r"] Nov 01 09:06:50.137600 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:06:50.137571 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76b035b_57e9_4f2b_bcc0_814437e63941.slice/crio-c06110835786aa19edeccbc82b7fffccbab9f02b017cf1d8716b3b10fc54b2aa WatchSource:0}: Error finding container c06110835786aa19edeccbc82b7fffccbab9f02b017cf1d8716b3b10fc54b2aa: Status 404 returned error can't find the container with id c06110835786aa19edeccbc82b7fffccbab9f02b017cf1d8716b3b10fc54b2aa Nov 01 09:06:50.286434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.286346 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr"] Nov 01 09:06:50.292869 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:06:50.292837 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf642b66_bd84_4012_a8d8_3a3eabc6878e.slice/crio-9d75431ff09126f2e919437bf993687501dd19937bc7d085164376d7096ca472 WatchSource:0}: Error finding container 9d75431ff09126f2e919437bf993687501dd19937bc7d085164376d7096ca472: Status 404 returned error can't find the container with id 9d75431ff09126f2e919437bf993687501dd19937bc7d085164376d7096ca472 Nov 01 09:06:50.983862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.983828 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" event={"ID":"df642b66-bd84-4012-a8d8-3a3eabc6878e","Type":"ContainerStarted","Data":"c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47"} Nov 01 09:06:50.984099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.983884 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" event={"ID":"df642b66-bd84-4012-a8d8-3a3eabc6878e","Type":"ContainerStarted","Data":"9d75431ff09126f2e919437bf993687501dd19937bc7d085164376d7096ca472"} Nov 01 09:06:50.984099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.983936 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:06:50.985746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.985705 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" event={"ID":"f76b035b-57e9-4f2b-bcc0-814437e63941","Type":"ContainerStarted","Data":"853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def"} Nov 01 09:06:50.985932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.985774 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:06:50.985932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:50.985800 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" event={"ID":"f76b035b-57e9-4f2b-bcc0-814437e63941","Type":"ContainerStarted","Data":"c06110835786aa19edeccbc82b7fffccbab9f02b017cf1d8716b3b10fc54b2aa"} Nov 01 09:06:51.061801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:51.061770 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda72c59-82cd-48a0-a17b-e75a0289a8ad" path="/var/lib/kubelet/pods/cda72c59-82cd-48a0-a17b-e75a0289a8ad/volumes" Nov 01 09:06:51.103357 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:51.103299 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" podStartSLOduration=2.103283238 podStartE2EDuration="2.103283238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:51.101037648 +0000 UTC m=+2034.189374321" watchObservedRunningTime="2026-11-01 09:06:51.103283238 +0000 UTC m=+2034.191619910" Nov 01 09:06:51.236739 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:51.236589 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" podStartSLOduration=2.2365661980000002 podStartE2EDuration="2.236566198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:06:51.222095278 +0000 UTC m=+2034.310431951" watchObservedRunningTime="2026-11-01 09:06:51.236566198 +0000 UTC m=+2034.324902874" Nov 01 09:06:57.040252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:57.040216 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:06:59.668053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:06:59.668013 5710 scope.go:122] "RemoveContainer" containerID="654f420acd8338e9e037af97e05919de2b7eb3fea6818b48c9b1c960ca0373d8" Nov 01 09:07:02.003580 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:02.003540 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:07:02.004131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:02.003627 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:07:07.780971 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.780934 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94"] Nov 01 09:07:07.787129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.787103 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:07.839698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.839662 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94"] Nov 01 09:07:07.874925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.874872 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:07.875197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.875178 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrvhc\" (UniqueName: \"kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc\") pod \"payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:07.882525 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.882495 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-095909\"" Nov 01 09:07:07.882691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.882556 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-095909-dockercfg-h5j4v\"" Nov 01 09:07:07.976167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.976117 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrvhc\" (UniqueName: \"kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc\") pod \"payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:07.976369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.976201 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:07.976922 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:07.976903 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:08.034653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.034566 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrvhc\" (UniqueName: \"kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc\") pod \"payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:08.101584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.101535 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:08.493153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.493114 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94"] Nov 01 09:07:08.499862 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:07:08.499827 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a163c1_ee6e_42b2_910e_81bf40f30097.slice/crio-fc82054d268a42bf66053e8d41ef3a322cfa2ada5bace9a2618834f3394bf887 WatchSource:0}: Error finding container fc82054d268a42bf66053e8d41ef3a322cfa2ada5bace9a2618834f3394bf887: Status 404 returned error can't find the container with id fc82054d268a42bf66053e8d41ef3a322cfa2ada5bace9a2618834f3394bf887 Nov 01 09:07:08.562099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.562068 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7"] Nov 01 09:07:08.567811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.567789 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.583774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.583745 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x\") pod \"payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.583892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.583863 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.619873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.619836 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7"] Nov 01 09:07:08.685504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.685464 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x\") pod \"payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.685753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.685559 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.686338 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.686311 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.743297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.743215 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x\") pod \"payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:08.887242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:08.887192 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:09.139532 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:09.139493 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" event={"ID":"03a163c1-ee6e-42b2-910e-81bf40f30097","Type":"ContainerStarted","Data":"b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913"} Nov 01 09:07:09.139769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:09.139558 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" event={"ID":"03a163c1-ee6e-42b2-910e-81bf40f30097","Type":"ContainerStarted","Data":"fc82054d268a42bf66053e8d41ef3a322cfa2ada5bace9a2618834f3394bf887"} Nov 01 09:07:09.139769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:09.139583 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:09.249923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:09.249869 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" podStartSLOduration=2.249852184 podStartE2EDuration="2.249852184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:09.24836306 +0000 UTC m=+2052.336699732" watchObservedRunningTime="2026-11-01 09:07:09.249852184 +0000 UTC m=+2052.338188856" Nov 01 09:07:09.264516 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:09.264463 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7"] Nov 01 09:07:09.271326 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:07:09.271297 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c9ac6c_1566_4f9d_8478_46fc13d615fa.slice/crio-114fea0ee9587238be7b9e7e38406f26cad8bf5f058044a2e1297f6dec5f1fb3 WatchSource:0}: Error finding container 114fea0ee9587238be7b9e7e38406f26cad8bf5f058044a2e1297f6dec5f1fb3: Status 404 returned error can't find the container with id 114fea0ee9587238be7b9e7e38406f26cad8bf5f058044a2e1297f6dec5f1fb3 Nov 01 09:07:10.148889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:10.148851 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" event={"ID":"10c9ac6c-1566-4f9d-8478-46fc13d615fa","Type":"ContainerStarted","Data":"1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b"} Nov 01 09:07:10.149383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:10.148910 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" event={"ID":"10c9ac6c-1566-4f9d-8478-46fc13d615fa","Type":"ContainerStarted","Data":"114fea0ee9587238be7b9e7e38406f26cad8bf5f058044a2e1297f6dec5f1fb3"} Nov 01 09:07:10.149383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:10.148947 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:10.264775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:10.264679 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" podStartSLOduration=2.264663373 podStartE2EDuration="2.264663373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:10.261911856 +0000 UTC m=+2053.350248539" watchObservedRunningTime="2026-11-01 09:07:10.264663373 +0000 UTC m=+2053.353000045" Nov 01 09:07:15.479598 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:07:15.479557 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:07:20.151825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:20.151795 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:07:21.157729 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:21.157681 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:07:38.779729 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:38.779620 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:07:39.840705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:39.840657 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf"] Nov 01 09:07:39.841245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:39.840935 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" podUID="3f64cd8e-2d20-48cf-aa9f-11373510dd28" containerName="maas-api" containerID="cri-o://c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880" gracePeriod=30 Nov 01 09:07:40.137406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.137372 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:07:40.234830 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.234384 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca\") pod \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " Nov 01 09:07:40.234830 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.234571 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc\" (UniqueName: \"kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc\") pod \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " Nov 01 09:07:40.234830 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.234798 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\") pod \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " Nov 01 09:07:40.235160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.234888 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs\") pod \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\" (UID: \"3f64cd8e-2d20-48cf-aa9f-11373510dd28\") " Nov 01 09:07:40.236030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.235993 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca" pod "3f64cd8e-2d20-48cf-aa9f-11373510dd28" (UID: "3f64cd8e-2d20-48cf-aa9f-11373510dd28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:07:40.249943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.249896 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls" pod "3f64cd8e-2d20-48cf-aa9f-11373510dd28" (UID: "3f64cd8e-2d20-48cf-aa9f-11373510dd28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:07:40.249943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.249918 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc" pod "3f64cd8e-2d20-48cf-aa9f-11373510dd28" (UID: "3f64cd8e-2d20-48cf-aa9f-11373510dd28"). InnerVolumeSpecName "kube-api-access-5lznc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:07:40.265988 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.265913 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs" pod "3f64cd8e-2d20-48cf-aa9f-11373510dd28" (UID: "3f64cd8e-2d20-48cf-aa9f-11373510dd28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:07:40.336892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.336845 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:07:40.336892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.336883 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f64cd8e-2d20-48cf-aa9f-11373510dd28-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:07:40.336892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.336894 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f64cd8e-2d20-48cf-aa9f-11373510dd28-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:07:40.337173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.336905 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lznc\" (UniqueName: \"kubernetes.io/projected/3f64cd8e-2d20-48cf-aa9f-11373510dd28-kube-api-access-5lznc\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:07:40.361517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.361418 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5"] Nov 01 09:07:40.361695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.361662 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" podUID="33799e41-a6e0-431e-ba21-ae3ccd50c8ba" containerName="payload-pre-processing" containerID="cri-o://77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951" gracePeriod=30 Nov 01 09:07:40.406192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.406153 5710 generic.go:356] "Generic (PLEG): container finished" podID="3f64cd8e-2d20-48cf-aa9f-11373510dd28" containerID="c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880" exitCode=0 Nov 01 09:07:40.406402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.406265 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" Nov 01 09:07:40.406402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.406273 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" event={"ID":"3f64cd8e-2d20-48cf-aa9f-11373510dd28","Type":"ContainerDied","Data":"c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880"} Nov 01 09:07:40.406402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.406338 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf" event={"ID":"3f64cd8e-2d20-48cf-aa9f-11373510dd28","Type":"ContainerDied","Data":"f46c574d2c4eb0b6d067af04df65a7379315dc4dd516ebdf903093d0e0de282b"} Nov 01 09:07:40.406402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.406367 5710 scope.go:122] "RemoveContainer" containerID="c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880" Nov 01 09:07:40.478406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.478379 5710 scope.go:122] "RemoveContainer" containerID="c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880" Nov 01 09:07:40.478869 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:07:40.478843 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880\": container with ID starting with c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880 not found: ID does not exist" containerID="c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880" Nov 01 09:07:40.478977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.478882 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880"} err="failed to get container status \"c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880\": rpc error: code = NotFound desc = could not find container \"c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880\": container with ID starting with c10f1a569e9ba6f52172ebc3203e9ead1d419346df5681c4a370106b5a31b880 not found: ID does not exist" Nov 01 09:07:40.584106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.584059 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf"] Nov 01 09:07:40.589346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.588496 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-23fa20-69df85c87f-tmhwf"] Nov 01 09:07:40.647424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.647394 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:07:40.742228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.742197 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h\" (UniqueName: \"kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h\") pod \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " Nov 01 09:07:40.742472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.742452 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume\") pod \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\" (UID: \"33799e41-a6e0-431e-ba21-ae3ccd50c8ba\") " Nov 01 09:07:40.742820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.742798 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume" pod "33799e41-a6e0-431e-ba21-ae3ccd50c8ba" (UID: "33799e41-a6e0-431e-ba21-ae3ccd50c8ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:07:40.743016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.743000 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:07:40.744501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.744481 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h" pod "33799e41-a6e0-431e-ba21-ae3ccd50c8ba" (UID: "33799e41-a6e0-431e-ba21-ae3ccd50c8ba"). InnerVolumeSpecName "kube-api-access-kjh2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:07:40.866293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:40.865958 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjh2h\" (UniqueName: \"kubernetes.io/projected/33799e41-a6e0-431e-ba21-ae3ccd50c8ba-kube-api-access-kjh2h\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:07:41.059817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.059709 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f64cd8e-2d20-48cf-aa9f-11373510dd28" path="/var/lib/kubelet/pods/3f64cd8e-2d20-48cf-aa9f-11373510dd28/volumes" Nov 01 09:07:41.415171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.415137 5710 generic.go:356] "Generic (PLEG): container finished" podID="33799e41-a6e0-431e-ba21-ae3ccd50c8ba" containerID="77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951" exitCode=0 Nov 01 09:07:41.415363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.415186 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" Nov 01 09:07:41.415363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.415201 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" event={"ID":"33799e41-a6e0-431e-ba21-ae3ccd50c8ba","Type":"ContainerDied","Data":"77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951"} Nov 01 09:07:41.415363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.415252 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5" event={"ID":"33799e41-a6e0-431e-ba21-ae3ccd50c8ba","Type":"ContainerDied","Data":"2ef077c3687d3ab4de6fdce27035ffc9fef944eb2f8804aada51e06099e625bb"} Nov 01 09:07:41.415363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.415274 5710 scope.go:122] "RemoveContainer" containerID="77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951" Nov 01 09:07:41.425287 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.425263 5710 scope.go:122] "RemoveContainer" containerID="77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951" Nov 01 09:07:41.425557 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:07:41.425540 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951\": container with ID starting with 77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951 not found: ID does not exist" containerID="77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951" Nov 01 09:07:41.425609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.425567 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951"} err="failed to get container status \"77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951\": rpc error: code = NotFound desc = could not find container \"77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951\": container with ID starting with 77276dbaffd9b61ad0745d04918235b96e498b1b579dccbdd06a629841597951 not found: ID does not exist" Nov 01 09:07:41.572466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.572428 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5"] Nov 01 09:07:41.618404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:41.618366 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-23fa20-55864dc45f-8gcd5"] Nov 01 09:07:43.066562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:43.066525 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33799e41-a6e0-431e-ba21-ae3ccd50c8ba" path="/var/lib/kubelet/pods/33799e41-a6e0-431e-ba21-ae3ccd50c8ba/volumes" Nov 01 09:07:45.479019 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:07:45.478981 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:07:47.212890 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.212851 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4"] Nov 01 09:07:47.213489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.213474 5710 cpu_manager.go:414] "containerMap: removing container" podUID="33799e41-a6e0-431e-ba21-ae3ccd50c8ba" containerName="payload-pre-processing" Nov 01 09:07:47.213553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.213490 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33799e41-a6e0-431e-ba21-ae3ccd50c8ba" containerName="payload-pre-processing" Nov 01 09:07:47.213553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.213507 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3f64cd8e-2d20-48cf-aa9f-11373510dd28" containerName="maas-api" Nov 01 09:07:47.213553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.213514 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f64cd8e-2d20-48cf-aa9f-11373510dd28" containerName="maas-api" Nov 01 09:07:47.213676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.213658 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="33799e41-a6e0-431e-ba21-ae3ccd50c8ba" containerName="payload-pre-processing" Nov 01 09:07:47.213676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.213668 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f64cd8e-2d20-48cf-aa9f-11373510dd28" containerName="maas-api" Nov 01 09:07:47.217317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.217294 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.271458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.271045 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.271740 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.271702 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4"] Nov 01 09:07:47.272013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.271976 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8t2t\" (UniqueName: \"kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.314658 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.314620 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b893c8\"" Nov 01 09:07:47.314870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.314745 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b893c8-dockercfg-n87k4\"" Nov 01 09:07:47.373404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.373224 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.373628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.373437 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8t2t\" (UniqueName: \"kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.374241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.374216 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.432114 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.432074 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8t2t\" (UniqueName: \"kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t\") pod \"payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.533970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.533871 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:47.918500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:47.918460 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4"] Nov 01 09:07:47.925838 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:07:47.925804 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bebc555_84e5_4f38_a390_58c1ccf4b027.slice/crio-81e9bd5ecfe5a19e83839f58de95bc8d4fae70d9b0386c849107a6341809d5f6 WatchSource:0}: Error finding container 81e9bd5ecfe5a19e83839f58de95bc8d4fae70d9b0386c849107a6341809d5f6: Status 404 returned error can't find the container with id 81e9bd5ecfe5a19e83839f58de95bc8d4fae70d9b0386c849107a6341809d5f6 Nov 01 09:07:48.011927 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.011892 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx"] Nov 01 09:07:48.016503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.016475 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.069962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.069919 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx"] Nov 01 09:07:48.082344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.082302 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwwq\" (UniqueName: \"kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq\") pod \"payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.082573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.082531 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.183889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.183784 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.184088 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.183947 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwwq\" (UniqueName: \"kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq\") pod \"payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.184556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.184526 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.242553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.242520 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwwq\" (UniqueName: \"kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq\") pod \"payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.332458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.332413 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:48.466436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.466320 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" event={"ID":"1bebc555-84e5-4f38-a390-58c1ccf4b027","Type":"ContainerStarted","Data":"332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919"} Nov 01 09:07:48.466436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.466430 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" event={"ID":"1bebc555-84e5-4f38-a390-58c1ccf4b027","Type":"ContainerStarted","Data":"81e9bd5ecfe5a19e83839f58de95bc8d4fae70d9b0386c849107a6341809d5f6"} Nov 01 09:07:48.466698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.466657 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:07:48.577167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.577107 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" podStartSLOduration=1.5770915749999999 podStartE2EDuration="1.577091575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:48.575827797 +0000 UTC m=+2091.664164470" watchObservedRunningTime="2026-11-01 09:07:48.577091575 +0000 UTC m=+2091.665428247" Nov 01 09:07:48.732797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:48.729704 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx"] Nov 01 09:07:48.732797 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:07:48.732088 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72205af_d592_4096_aba0_24efe89c01ee.slice/crio-4cf99ffb9857e91bb0721bd17162961579a0f77c0c49948fb7bc68f2aae21633 WatchSource:0}: Error finding container 4cf99ffb9857e91bb0721bd17162961579a0f77c0c49948fb7bc68f2aae21633: Status 404 returned error can't find the container with id 4cf99ffb9857e91bb0721bd17162961579a0f77c0c49948fb7bc68f2aae21633 Nov 01 09:07:49.478265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:49.478214 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" event={"ID":"d72205af-d592-4096-aba0-24efe89c01ee","Type":"ContainerStarted","Data":"10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b"} Nov 01 09:07:49.478868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:49.478281 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" event={"ID":"d72205af-d592-4096-aba0-24efe89c01ee","Type":"ContainerStarted","Data":"4cf99ffb9857e91bb0721bd17162961579a0f77c0c49948fb7bc68f2aae21633"} Nov 01 09:07:49.478868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:49.478312 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:07:49.591710 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:49.591647 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" podStartSLOduration=2.591630293 podStartE2EDuration="2.591630293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:07:49.588638567 +0000 UTC m=+2092.676975253" watchObservedRunningTime="2026-11-01 09:07:49.591630293 +0000 UTC m=+2092.679966965" Nov 01 09:07:59.485393 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:07:59.483912 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:08:00.486914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:00.486882 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:08:00.912134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:00.912108 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:08:00.924551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:00.924521 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:08:08.669784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.661395 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt"] Nov 01 09:08:08.681650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.681612 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:08.705639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.705604 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt"] Nov 01 09:08:08.717790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.716252 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:08.717790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.716415 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8ml\" (UniqueName: \"kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml\") pod \"payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:08.748456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.748427 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5fe64d\"" Nov 01 09:08:08.748695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.748499 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5fe64d-dockercfg-pwr7w\"" Nov 01 09:08:08.818412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.818367 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:08.818640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.818431 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8ml\" (UniqueName: \"kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml\") pod \"payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:08.819178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.819157 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:08.876693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:08.876648 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8ml\" (UniqueName: \"kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml\") pod \"payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:09.000486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.000399 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:09.021168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.021122 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk"] Nov 01 09:08:09.021431 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.021407 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" podUID="61c26a0c-731c-4593-b1ef-11192fae18fe" containerName="maas-api" containerID="cri-o://e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed" gracePeriod=30 Nov 01 09:08:09.337104 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.337076 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:08:09.440445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.436521 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt"] Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.456788 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs\" (UniqueName: \"kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs\") pod \"61c26a0c-731c-4593-b1ef-11192fae18fe\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.456928 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs\") pod \"61c26a0c-731c-4593-b1ef-11192fae18fe\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.457014 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca\") pod \"61c26a0c-731c-4593-b1ef-11192fae18fe\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.457085 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\") pod \"61c26a0c-731c-4593-b1ef-11192fae18fe\" (UID: \"61c26a0c-731c-4593-b1ef-11192fae18fe\") " Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.458346 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca" pod "61c26a0c-731c-4593-b1ef-11192fae18fe" (UID: "61c26a0c-731c-4593-b1ef-11192fae18fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.461423 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls" pod "61c26a0c-731c-4593-b1ef-11192fae18fe" (UID: "61c26a0c-731c-4593-b1ef-11192fae18fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.464983 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs" pod "61c26a0c-731c-4593-b1ef-11192fae18fe" (UID: "61c26a0c-731c-4593-b1ef-11192fae18fe"). InnerVolumeSpecName "kube-api-access-kwtjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:09.469653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.466855 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs" pod "61c26a0c-731c-4593-b1ef-11192fae18fe" (UID: "61c26a0c-731c-4593-b1ef-11192fae18fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:08:09.491553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.491515 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5"] Nov 01 09:08:09.491842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.491810 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" podUID="28af8d2e-8809-430f-a082-006b0087b4fe" containerName="payload-processing" containerID="cri-o://915c38cc2a470e6a9ef12cdc47a5094dda198818962618168700ca44bde35bbd" gracePeriod=30 Nov 01 09:08:09.558693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.558663 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:09.558693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.558696 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c26a0c-731c-4593-b1ef-11192fae18fe-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:09.558907 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.558710 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c26a0c-731c-4593-b1ef-11192fae18fe-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:09.558907 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.558759 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtjs\" (UniqueName: \"kubernetes.io/projected/61c26a0c-731c-4593-b1ef-11192fae18fe-kube-api-access-kwtjs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:09.762524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.762487 5710 generic.go:356] "Generic (PLEG): container finished" podID="28af8d2e-8809-430f-a082-006b0087b4fe" containerID="915c38cc2a470e6a9ef12cdc47a5094dda198818962618168700ca44bde35bbd" exitCode=0 Nov 01 09:08:09.763046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.762564 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" event={"ID":"28af8d2e-8809-430f-a082-006b0087b4fe","Type":"ContainerDied","Data":"915c38cc2a470e6a9ef12cdc47a5094dda198818962618168700ca44bde35bbd"} Nov 01 09:08:09.763046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.762623 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:08:09.763046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.762635 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" event={"ID":"28af8d2e-8809-430f-a082-006b0087b4fe","Type":"ContainerDied","Data":"71ede97e0a330f4ddbea3521c61650bc649b4e1e1100c318bcca7efc54b91c44"} Nov 01 09:08:09.763046 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.762662 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ede97e0a330f4ddbea3521c61650bc649b4e1e1100c318bcca7efc54b91c44" Nov 01 09:08:09.764545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.764515 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" event={"ID":"b4628121-3162-4d72-a705-1f815ca00a56","Type":"ContainerStarted","Data":"8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd"} Nov 01 09:08:09.764704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.764574 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" event={"ID":"b4628121-3162-4d72-a705-1f815ca00a56","Type":"ContainerStarted","Data":"277ebb382f7a484e101b761397c732f9bd2fa8df3fd326ebd5f56cb72c7c6ad9"} Nov 01 09:08:09.764704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.764604 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:09.766083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.766054 5710 generic.go:356] "Generic (PLEG): container finished" podID="61c26a0c-731c-4593-b1ef-11192fae18fe" containerID="e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed" exitCode=0 Nov 01 09:08:09.766192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.766105 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" Nov 01 09:08:09.766192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.766134 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" event={"ID":"61c26a0c-731c-4593-b1ef-11192fae18fe","Type":"ContainerDied","Data":"e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed"} Nov 01 09:08:09.766192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.766184 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk" event={"ID":"61c26a0c-731c-4593-b1ef-11192fae18fe","Type":"ContainerDied","Data":"5d8772e9c091020aa73e3f9c177afc0c9ca2e315515ddc804f9cf9af952b7460"} Nov 01 09:08:09.766306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.766208 5710 scope.go:122] "RemoveContainer" containerID="e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed" Nov 01 09:08:09.779562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.779539 5710 scope.go:122] "RemoveContainer" containerID="e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed" Nov 01 09:08:09.779984 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:09.779962 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed\": container with ID starting with e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed not found: ID does not exist" containerID="e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed" Nov 01 09:08:09.780048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.779995 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed"} err="failed to get container status \"e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed\": rpc error: code = NotFound desc = could not find container \"e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed\": container with ID starting with e49ed90a7fe4ee6a5c815584750988713afec59822c06348780f6913d37b60ed not found: ID does not exist" Nov 01 09:08:09.862452 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.862411 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm\" (UniqueName: \"kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm\") pod \"28af8d2e-8809-430f-a082-006b0087b4fe\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " Nov 01 09:08:09.862677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.862517 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume\") pod \"28af8d2e-8809-430f-a082-006b0087b4fe\" (UID: \"28af8d2e-8809-430f-a082-006b0087b4fe\") " Nov 01 09:08:09.863038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.863009 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume" pod "28af8d2e-8809-430f-a082-006b0087b4fe" (UID: "28af8d2e-8809-430f-a082-006b0087b4fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:09.864818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.864794 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm" pod "28af8d2e-8809-430f-a082-006b0087b4fe" (UID: "28af8d2e-8809-430f-a082-006b0087b4fe"). InnerVolumeSpecName "kube-api-access-fmqlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:09.963997 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.963962 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28af8d2e-8809-430f-a082-006b0087b4fe-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:09.963997 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.963991 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmqlm\" (UniqueName: \"kubernetes.io/projected/28af8d2e-8809-430f-a082-006b0087b4fe-kube-api-access-fmqlm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:09.984243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:09.984180 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" podStartSLOduration=1.984163854 podStartE2EDuration="1.984163854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:09.982325355 +0000 UTC m=+2113.070662029" watchObservedRunningTime="2026-11-01 09:08:09.984163854 +0000 UTC m=+2113.072500525" Nov 01 09:08:10.095489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:10.095449 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk"] Nov 01 09:08:10.101858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:10.101825 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73660a-5cdc6b4b57-ql7tk"] Nov 01 09:08:10.775165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:10.775121 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5" Nov 01 09:08:10.930416 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:10.930379 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5"] Nov 01 09:08:10.976760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:10.976700 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73660a-546bc77cb8-zhgb5"] Nov 01 09:08:11.059802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:11.059697 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28af8d2e-8809-430f-a082-006b0087b4fe" path="/var/lib/kubelet/pods/28af8d2e-8809-430f-a082-006b0087b4fe/volumes" Nov 01 09:08:11.060237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:11.060211 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c26a0c-731c-4593-b1ef-11192fae18fe" path="/var/lib/kubelet/pods/61c26a0c-731c-4593-b1ef-11192fae18fe/volumes" Nov 01 09:08:12.853966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:12.853933 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:08:15.479510 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:15.479468 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:08:18.850536 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:18.850499 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c"] Nov 01 09:08:18.851045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:18.850785 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" containerID="cri-o://e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d" gracePeriod=30 Nov 01 09:08:20.777855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:20.777824 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:08:24.127769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.127692 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:08:24.261455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261411 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261462 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261520 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261563 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261595 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261649 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261769 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:24.261992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261787 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261871 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.261992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.261910 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d\") pod \"ba49da73-1f72-46fd-affa-a286031ead32\" (UID: \"ba49da73-1f72-46fd-affa-a286031ead32\") " Nov 01 09:08:24.262195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262009 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:24.262195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262098 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:24.262195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262102 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:24.262406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262357 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:24.262651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262628 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.262755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262657 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.262755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262672 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.262755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262687 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba49da73-1f72-46fd-affa-a286031ead32-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.262755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.262701 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.264105 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.264083 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:08:24.264204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.264154 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:24.264204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.264188 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:24.264297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.264282 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d" pod "ba49da73-1f72-46fd-affa-a286031ead32" (UID: "ba49da73-1f72-46fd-affa-a286031ead32"). InnerVolumeSpecName "kube-api-access-lrv9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:24.364132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.364094 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba49da73-1f72-46fd-affa-a286031ead32-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.364132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.364127 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba49da73-1f72-46fd-affa-a286031ead32-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.364132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.364139 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrv9d\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-kube-api-access-lrv9d\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.364394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.364149 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba49da73-1f72-46fd-affa-a286031ead32-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:24.890952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.890915 5710 generic.go:356] "Generic (PLEG): container finished" podID="ba49da73-1f72-46fd-affa-a286031ead32" containerID="e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d" exitCode=0 Nov 01 09:08:24.891144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.890978 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" Nov 01 09:08:24.891144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.890998 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" event={"ID":"ba49da73-1f72-46fd-affa-a286031ead32","Type":"ContainerDied","Data":"e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d"} Nov 01 09:08:24.891144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.891086 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c" event={"ID":"ba49da73-1f72-46fd-affa-a286031ead32","Type":"ContainerDied","Data":"2349b4f990deabf94a9ac9d07ae8d7a1501dba0f934c63475cfa54fa81c948e4"} Nov 01 09:08:24.891144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.891110 5710 scope.go:122] "RemoveContainer" containerID="e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d" Nov 01 09:08:24.903826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.903803 5710 scope.go:122] "RemoveContainer" containerID="f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094" Nov 01 09:08:24.914836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.914808 5710 scope.go:122] "RemoveContainer" containerID="e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d" Nov 01 09:08:24.915249 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:24.915227 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d\": container with ID starting with e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d not found: ID does not exist" containerID="e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d" Nov 01 09:08:24.915297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.915263 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d"} err="failed to get container status \"e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d\": rpc error: code = NotFound desc = could not find container \"e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d\": container with ID starting with e91beb0cc70bc3b2d0bd6275d44f47a093112037f19619eed9068195f46eb58d not found: ID does not exist" Nov 01 09:08:24.915297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.915284 5710 scope.go:122] "RemoveContainer" containerID="f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094" Nov 01 09:08:24.915630 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:24.915612 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094\": container with ID starting with f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094 not found: ID does not exist" containerID="f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094" Nov 01 09:08:24.915676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:24.915639 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094"} err="failed to get container status \"f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094\": rpc error: code = NotFound desc = could not find container \"f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094\": container with ID starting with f2c48543922ca6a226fac6965c86b0ed499bb41468c7dc85a51cdc614a93e094 not found: ID does not exist" Nov 01 09:08:25.045618 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:25.045584 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c"] Nov 01 09:08:25.092314 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:25.092274 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-73660a-openshift-default-59cc5cbb7c-wwj6c"] Nov 01 09:08:27.059154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:27.059116 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba49da73-1f72-46fd-affa-a286031ead32" path="/var/lib/kubelet/pods/ba49da73-1f72-46fd-affa-a286031ead32/volumes" Nov 01 09:08:34.126127 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:34.126091 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk"] Nov 01 09:08:34.206765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:34.206686 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg"] Nov 01 09:08:39.126986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.126916 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" podUID="220dd668-434b-445a-92ab-a9846e908061" containerName="payload-processing" containerID="cri-o://2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc" gracePeriod=25 Nov 01 09:08:39.208062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.208013 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" podUID="6abe3ec1-9c61-412e-b1d1-149552dd46b5" containerName="payload-pre-processing" containerID="cri-o://145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158" gracePeriod=25 Nov 01 09:08:39.252180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.252136 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:08:39.499626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.499598 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:08:39.523488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.523458 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:08:39.557967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.557919 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2\" (UniqueName: \"kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2\") pod \"220dd668-434b-445a-92ab-a9846e908061\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " Nov 01 09:08:39.558134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558064 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume\") pod \"220dd668-434b-445a-92ab-a9846e908061\" (UID: \"220dd668-434b-445a-92ab-a9846e908061\") " Nov 01 09:08:39.558204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558130 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz\" (UniqueName: \"kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz\") pod \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " Nov 01 09:08:39.558204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558192 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume\") pod \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\" (UID: \"6abe3ec1-9c61-412e-b1d1-149552dd46b5\") " Nov 01 09:08:39.558480 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558450 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume" pod "220dd668-434b-445a-92ab-a9846e908061" (UID: "220dd668-434b-445a-92ab-a9846e908061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:39.558623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558596 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume" pod "6abe3ec1-9c61-412e-b1d1-149552dd46b5" (UID: "6abe3ec1-9c61-412e-b1d1-149552dd46b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:39.558778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558760 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/220dd668-434b-445a-92ab-a9846e908061-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:39.558835 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.558781 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abe3ec1-9c61-412e-b1d1-149552dd46b5-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:39.560588 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.560564 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz" pod "6abe3ec1-9c61-412e-b1d1-149552dd46b5" (UID: "6abe3ec1-9c61-412e-b1d1-149552dd46b5"). InnerVolumeSpecName "kube-api-access-8lhnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:39.560691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.560604 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2" pod "220dd668-434b-445a-92ab-a9846e908061" (UID: "220dd668-434b-445a-92ab-a9846e908061"). InnerVolumeSpecName "kube-api-access-wmzc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:39.660515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.660417 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzc2\" (UniqueName: \"kubernetes.io/projected/220dd668-434b-445a-92ab-a9846e908061-kube-api-access-wmzc2\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:39.660515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.660453 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhnz\" (UniqueName: \"kubernetes.io/projected/6abe3ec1-9c61-412e-b1d1-149552dd46b5-kube-api-access-8lhnz\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:39.991109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.991018 5710 generic.go:356] "Generic (PLEG): container finished" podID="6abe3ec1-9c61-412e-b1d1-149552dd46b5" containerID="145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158" exitCode=0 Nov 01 09:08:39.991109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.991083 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" Nov 01 09:08:39.991109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.991106 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" event={"ID":"6abe3ec1-9c61-412e-b1d1-149552dd46b5","Type":"ContainerDied","Data":"145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158"} Nov 01 09:08:39.991453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.991195 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg" event={"ID":"6abe3ec1-9c61-412e-b1d1-149552dd46b5","Type":"ContainerDied","Data":"64801bdf06e33a8b1d6882c6a303c8ce0705f3feb9a12db2cb78127d9cab7fac"} Nov 01 09:08:39.991453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.991240 5710 scope.go:122] "RemoveContainer" containerID="145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158" Nov 01 09:08:39.992875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.992792 5710 generic.go:356] "Generic (PLEG): container finished" podID="220dd668-434b-445a-92ab-a9846e908061" containerID="2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc" exitCode=0 Nov 01 09:08:39.992875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.992858 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" Nov 01 09:08:39.992875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.992852 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" event={"ID":"220dd668-434b-445a-92ab-a9846e908061","Type":"ContainerDied","Data":"2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc"} Nov 01 09:08:39.993242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:39.992966 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk" event={"ID":"220dd668-434b-445a-92ab-a9846e908061","Type":"ContainerDied","Data":"950b56071dd7cbc25944b9e5b0a4b3de941c1aa1e2159df9d08259f2b95bf196"} Nov 01 09:08:40.003336 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.003312 5710 scope.go:122] "RemoveContainer" containerID="145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158" Nov 01 09:08:40.003761 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:40.003700 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158\": container with ID starting with 145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158 not found: ID does not exist" containerID="145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158" Nov 01 09:08:40.003843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.003770 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158"} err="failed to get container status \"145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158\": rpc error: code = NotFound desc = could not find container \"145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158\": container with ID starting with 145891e11eb113b0bed94cd7dfee7e363479bb1c7d1684a5ec117eb7fb4bf158 not found: ID does not exist" Nov 01 09:08:40.003843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.003791 5710 scope.go:122] "RemoveContainer" containerID="2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc" Nov 01 09:08:40.014003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.013982 5710 scope.go:122] "RemoveContainer" containerID="2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc" Nov 01 09:08:40.014337 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:40.014317 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc\": container with ID starting with 2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc not found: ID does not exist" containerID="2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc" Nov 01 09:08:40.014410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.014348 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc"} err="failed to get container status \"2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc\": rpc error: code = NotFound desc = could not find container \"2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc\": container with ID starting with 2ecf39fc87aa2d8ac4dd34dea77c235e05ebe9cc5c45a5723e1acd4bbb0ab4fc not found: ID does not exist" Nov 01 09:08:40.173391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.173358 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg"] Nov 01 09:08:40.221454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.221407 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0460d-557c4d4595-gqffg"] Nov 01 09:08:40.322189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.322104 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk"] Nov 01 09:08:40.326237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:40.326207 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0460d-84464fcdbf-kwvkk"] Nov 01 09:08:41.059015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:41.058981 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220dd668-434b-445a-92ab-a9846e908061" path="/var/lib/kubelet/pods/220dd668-434b-445a-92ab-a9846e908061/volumes" Nov 01 09:08:41.059454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:41.059440 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abe3ec1-9c61-412e-b1d1-149552dd46b5" path="/var/lib/kubelet/pods/6abe3ec1-9c61-412e-b1d1-149552dd46b5/volumes" Nov 01 09:08:42.463260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.463213 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876"] Nov 01 09:08:42.463691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.463445 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" podUID="aa543cb9-c56e-436c-8610-1db7598247bf" containerName="maas-api" containerID="cri-o://76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34" gracePeriod=30 Nov 01 09:08:42.795440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.795412 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:08:42.907429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.907396 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr\" (UniqueName: \"kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr\") pod \"aa543cb9-c56e-436c-8610-1db7598247bf\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " Nov 01 09:08:42.907660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.907456 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs\") pod \"aa543cb9-c56e-436c-8610-1db7598247bf\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " Nov 01 09:08:42.907660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.907484 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca\") pod \"aa543cb9-c56e-436c-8610-1db7598247bf\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " Nov 01 09:08:42.907660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.907647 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\") pod \"aa543cb9-c56e-436c-8610-1db7598247bf\" (UID: \"aa543cb9-c56e-436c-8610-1db7598247bf\") " Nov 01 09:08:42.908205 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.908163 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca" pod "aa543cb9-c56e-436c-8610-1db7598247bf" (UID: "aa543cb9-c56e-436c-8610-1db7598247bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:42.908349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.908328 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa543cb9-c56e-436c-8610-1db7598247bf-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:42.910269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.910239 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr" pod "aa543cb9-c56e-436c-8610-1db7598247bf" (UID: "aa543cb9-c56e-436c-8610-1db7598247bf"). InnerVolumeSpecName "kube-api-access-r4vzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:42.910377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.910314 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs" pod "aa543cb9-c56e-436c-8610-1db7598247bf" (UID: "aa543cb9-c56e-436c-8610-1db7598247bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:08:42.914543 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:42.914520 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls" pod "aa543cb9-c56e-436c-8610-1db7598247bf" (UID: "aa543cb9-c56e-436c-8610-1db7598247bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:08:43.010203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.010137 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4vzr\" (UniqueName: \"kubernetes.io/projected/aa543cb9-c56e-436c-8610-1db7598247bf-kube-api-access-r4vzr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:43.010203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.010180 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:43.010203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.010195 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa543cb9-c56e-436c-8610-1db7598247bf-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:43.016789 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.016673 5710 generic.go:356] "Generic (PLEG): container finished" podID="aa543cb9-c56e-436c-8610-1db7598247bf" containerID="76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34" exitCode=0 Nov 01 09:08:43.016941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.016801 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" event={"ID":"aa543cb9-c56e-436c-8610-1db7598247bf","Type":"ContainerDied","Data":"76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34"} Nov 01 09:08:43.016941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.016855 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" event={"ID":"aa543cb9-c56e-436c-8610-1db7598247bf","Type":"ContainerDied","Data":"c599a00a473d812d9167a6ee03928acc702462eab46f8b3deea07aa46964ac9a"} Nov 01 09:08:43.016941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.016885 5710 scope.go:122] "RemoveContainer" containerID="76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34" Nov 01 09:08:43.017074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.017002 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876" Nov 01 09:08:43.041973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.041949 5710 scope.go:122] "RemoveContainer" containerID="76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34" Nov 01 09:08:43.043469 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:43.043442 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34\": container with ID starting with 76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34 not found: ID does not exist" containerID="76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34" Nov 01 09:08:43.043576 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.043477 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34"} err="failed to get container status \"76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34\": rpc error: code = NotFound desc = could not find container \"76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34\": container with ID starting with 76de42786ecb6aec13a4fbbdbb24ff68a85802318038f5d46459758e1202df34 not found: ID does not exist" Nov 01 09:08:43.172345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.172311 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876"] Nov 01 09:08:43.176218 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:43.176184 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eb28ec-66ff4d68fd-n2876"] Nov 01 09:08:45.061921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:45.059700 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa543cb9-c56e-436c-8610-1db7598247bf" path="/var/lib/kubelet/pods/aa543cb9-c56e-436c-8610-1db7598247bf/volumes" Nov 01 09:08:45.479033 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:45.478998 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:08:51.398639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:51.398602 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk"] Nov 01 09:08:51.399121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:51.398867 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" containerID="cri-o://adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36" gracePeriod=30 Nov 01 09:08:54.624911 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.624862 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww"] Nov 01 09:08:54.625503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625489 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" Nov 01 09:08:54.625503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625504 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625519 5710 cpu_manager.go:414] "containerMap: removing container" podUID="28af8d2e-8809-430f-a082-006b0087b4fe" containerName="payload-processing" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625525 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28af8d2e-8809-430f-a082-006b0087b4fe" containerName="payload-processing" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625534 5710 cpu_manager.go:414] "containerMap: removing container" podUID="aa543cb9-c56e-436c-8610-1db7598247bf" containerName="maas-api" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625538 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa543cb9-c56e-436c-8610-1db7598247bf" containerName="maas-api" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625566 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6abe3ec1-9c61-412e-b1d1-149552dd46b5" containerName="payload-pre-processing" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625573 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abe3ec1-9c61-412e-b1d1-149552dd46b5" containerName="payload-pre-processing" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625586 5710 cpu_manager.go:414] "containerMap: removing container" podUID="220dd668-434b-445a-92ab-a9846e908061" containerName="payload-processing" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625594 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220dd668-434b-445a-92ab-a9846e908061" containerName="payload-processing" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625616 5710 cpu_manager.go:414] "containerMap: removing container" podUID="61c26a0c-731c-4593-b1ef-11192fae18fe" containerName="maas-api" Nov 01 09:08:54.625621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625621 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c26a0c-731c-4593-b1ef-11192fae18fe" containerName="maas-api" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625839 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625856 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="28af8d2e-8809-430f-a082-006b0087b4fe" containerName="payload-processing" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625893 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa543cb9-c56e-436c-8610-1db7598247bf" containerName="maas-api" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625908 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abe3ec1-9c61-412e-b1d1-149552dd46b5" containerName="payload-pre-processing" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625918 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="220dd668-434b-445a-92ab-a9846e908061" containerName="payload-processing" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625929 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" Nov 01 09:08:54.626324 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.625938 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c26a0c-731c-4593-b1ef-11192fae18fe" containerName="maas-api" Nov 01 09:08:54.630026 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.630003 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.685939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.685894 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c3aaa7-openshift-default-dockercfg-bdpxt\"" Nov 01 09:08:54.746282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.746239 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww"] Nov 01 09:08:54.787473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.787429 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.787668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.787489 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.787668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.787567 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.787668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.787598 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.787668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.787648 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.791600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.790825 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.791600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.790952 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.791600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.791022 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.791600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.791081 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892113 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892173 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892257 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892288 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892343 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892366 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892453 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892519 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892577 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892589 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.892851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.892637 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.893078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.893049 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.893350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.893324 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.893486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.893467 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.895475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.895449 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.896059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.896034 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.965422 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.965342 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:54.969840 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:54.969560 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl\") pod \"e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:55.244779 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:55.244659 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:55.678003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:55.677956 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww"] Nov 01 09:08:55.685702 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:08:55.685668 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fd7f11_48e2_4703_af42_22d29b0cd125.slice/crio-72899e73785fa66ea083eb7296383aeb0768e889de6f71424a564602f4d36125 WatchSource:0}: Error finding container 72899e73785fa66ea083eb7296383aeb0768e889de6f71424a564602f4d36125: Status 404 returned error can't find the container with id 72899e73785fa66ea083eb7296383aeb0768e889de6f71424a564602f4d36125 Nov 01 09:08:55.688699 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:55.688662 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:08:55.688842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:55.688778 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:08:55.688885 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:55.688865 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:08:56.128743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.123641 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" event={"ID":"63fd7f11-48e2-4703-af42-22d29b0cd125","Type":"ContainerStarted","Data":"e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202"} Nov 01 09:08:56.128743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.123735 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" event={"ID":"63fd7f11-48e2-4703-af42-22d29b0cd125","Type":"ContainerStarted","Data":"72899e73785fa66ea083eb7296383aeb0768e889de6f71424a564602f4d36125"} Nov 01 09:08:56.245048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.245007 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:56.252031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.251989 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:56.451838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.450625 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podStartSLOduration=2.450601894 podStartE2EDuration="2.450601894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:56.313610172 +0000 UTC m=+2159.401946855" watchObservedRunningTime="2026-11-01 09:08:56.450601894 +0000 UTC m=+2159.538938567" Nov 01 09:08:56.679680 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.679603 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:08:56.722427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722387 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722509 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722533 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722554 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722598 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722667 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:56.722928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722739 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722784 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722811 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.722928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.722852 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert\") pod \"462e1c7c-0c60-4311-a492-de0082e58c32\" (UID: \"462e1c7c-0c60-4311-a492-de0082e58c32\") " Nov 01 09:08:56.723204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723017 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:56.723204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723126 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:56.723325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723206 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:56.723325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723293 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:08:56.723759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723704 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.723759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723750 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.723915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723765 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.723915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723780 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.723915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.723794 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462e1c7c-0c60-4311-a492-de0082e58c32-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.725120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.725094 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:08:56.725232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.725176 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:08:56.725496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.725476 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:56.725593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.725574 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb" pod "462e1c7c-0c60-4311-a492-de0082e58c32" (UID: "462e1c7c-0c60-4311-a492-de0082e58c32"). InnerVolumeSpecName "kube-api-access-2qlbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:08:56.825054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.825010 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462e1c7c-0c60-4311-a492-de0082e58c32-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.825054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.825051 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlbb\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-kube-api-access-2qlbb\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.825054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.825066 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462e1c7c-0c60-4311-a492-de0082e58c32-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.825320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.825078 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462e1c7c-0c60-4311-a492-de0082e58c32-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:08:56.983698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.983623 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv"] Nov 01 09:08:56.984331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984308 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" Nov 01 09:08:56.984432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984335 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49da73-1f72-46fd-affa-a286031ead32" containerName="istio-proxy" Nov 01 09:08:56.984432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984356 5710 cpu_manager.go:414] "containerMap: removing container" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" Nov 01 09:08:56.984432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984364 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" Nov 01 09:08:56.984432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984403 5710 cpu_manager.go:414] "containerMap: removing container" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" Nov 01 09:08:56.984432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984411 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" Nov 01 09:08:56.984595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.984553 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" Nov 01 09:08:56.988173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:56.988155 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.026638 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.026603 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prj8\" (UniqueName: \"kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.026866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.026678 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.038020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.037970 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eaabf4\"" Nov 01 09:08:57.038957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.038928 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eaabf4-dockercfg-ghn44\"" Nov 01 09:08:57.042337 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.042296 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv"] Nov 01 09:08:57.127930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.127888 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.128134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.128049 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prj8\" (UniqueName: \"kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.128681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.128655 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.132548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.132520 5710 generic.go:356] "Generic (PLEG): container finished" podID="462e1c7c-0c60-4311-a492-de0082e58c32" containerID="adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36" exitCode=0 Nov 01 09:08:57.132676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.132572 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" Nov 01 09:08:57.132676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.132600 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" event={"ID":"462e1c7c-0c60-4311-a492-de0082e58c32","Type":"ContainerDied","Data":"adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36"} Nov 01 09:08:57.132676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.132657 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk" event={"ID":"462e1c7c-0c60-4311-a492-de0082e58c32","Type":"ContainerDied","Data":"0e8c8571ca84367d94f8ff1fa6cc560dfe1ac696ad5cd0df85ddb842b22a40d6"} Nov 01 09:08:57.132795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.132681 5710 scope.go:122] "RemoveContainer" containerID="adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36" Nov 01 09:08:57.133736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.133036 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:57.134161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.134132 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:08:57.143332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.143311 5710 scope.go:122] "RemoveContainer" containerID="036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f" Nov 01 09:08:57.153562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.153540 5710 scope.go:122] "RemoveContainer" containerID="adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36" Nov 01 09:08:57.154122 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:57.154096 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36\": container with ID starting with adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36 not found: ID does not exist" containerID="adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36" Nov 01 09:08:57.154233 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.154130 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36"} err="failed to get container status \"adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36\": rpc error: code = NotFound desc = could not find container \"adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36\": container with ID starting with adb726755db02674d6745b2ab1899e8fa178df037c36573b71d3158053390e36 not found: ID does not exist" Nov 01 09:08:57.154233 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.154152 5710 scope.go:122] "RemoveContainer" containerID="036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f" Nov 01 09:08:57.154484 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:08:57.154462 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f\": container with ID starting with 036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f not found: ID does not exist" containerID="036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f" Nov 01 09:08:57.154529 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.154495 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f"} err="failed to get container status \"036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f\": rpc error: code = NotFound desc = could not find container \"036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f\": container with ID starting with 036c096a4838899a8f634994e4ca4fba67e06b8ca4c318b79391ab198fdd5b0f not found: ID does not exist" Nov 01 09:08:57.194491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.194460 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prj8\" (UniqueName: \"kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8\") pod \"payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.255084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.255001 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk"] Nov 01 09:08:57.259779 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.259746 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb28ec-openshift-default-6c99586f7f-6d4wk"] Nov 01 09:08:57.351059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.351025 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eaabf4-dockercfg-ghn44\"" Nov 01 09:08:57.353736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.353694 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:57.742866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.742830 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv"] Nov 01 09:08:57.750355 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:08:57.750328 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15350a6_9f01_49e9_a850_c210c8daaa2d.slice/crio-c936a9904f843f18912e31b1c4ce486acedcb5e23416c9cd22c77570e79881e9 WatchSource:0}: Error finding container c936a9904f843f18912e31b1c4ce486acedcb5e23416c9cd22c77570e79881e9: Status 404 returned error can't find the container with id c936a9904f843f18912e31b1c4ce486acedcb5e23416c9cd22c77570e79881e9 Nov 01 09:08:57.824953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.824923 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68"] Nov 01 09:08:57.825777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.825756 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" containerName="istio-proxy" Nov 01 09:08:57.829297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.829274 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:57.883712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.883677 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68"] Nov 01 09:08:57.946765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.946695 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzhj\" (UniqueName: \"kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj\") pod \"payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:57.946958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:57.946824 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:58.048325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.048230 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:58.048509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.048467 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzhj\" (UniqueName: \"kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj\") pod \"payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:58.048984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.048965 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:58.106289 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.106253 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzhj\" (UniqueName: \"kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj\") pod \"payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:58.141583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.141539 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" event={"ID":"f15350a6-9f01-49e9-a850-c210c8daaa2d","Type":"ContainerStarted","Data":"e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca"} Nov 01 09:08:58.141846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.141614 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" event={"ID":"f15350a6-9f01-49e9-a850-c210c8daaa2d","Type":"ContainerStarted","Data":"c936a9904f843f18912e31b1c4ce486acedcb5e23416c9cd22c77570e79881e9"} Nov 01 09:08:58.141846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.141647 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:08:58.144369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.144341 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:58.253910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.253856 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" podStartSLOduration=2.253842653 podStartE2EDuration="2.253842653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:58.250792663 +0000 UTC m=+2161.339129336" watchObservedRunningTime="2026-11-01 09:08:58.253842653 +0000 UTC m=+2161.342179352" Nov 01 09:08:58.527239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:58.527194 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68"] Nov 01 09:08:58.532940 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:08:58.532898 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346d880e_ef45_406b_9a77_80760c094c37.slice/crio-70f185a1bd0ec33963b485ac7cae27c2a7286ebafee10cc645e7b3f9a28f5d17 WatchSource:0}: Error finding container 70f185a1bd0ec33963b485ac7cae27c2a7286ebafee10cc645e7b3f9a28f5d17: Status 404 returned error can't find the container with id 70f185a1bd0ec33963b485ac7cae27c2a7286ebafee10cc645e7b3f9a28f5d17 Nov 01 09:08:59.061155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:59.061117 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462e1c7c-0c60-4311-a492-de0082e58c32" path="/var/lib/kubelet/pods/462e1c7c-0c60-4311-a492-de0082e58c32/volumes" Nov 01 09:08:59.151824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:59.151782 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" event={"ID":"346d880e-ef45-406b-9a77-80760c094c37","Type":"ContainerStarted","Data":"de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09"} Nov 01 09:08:59.152005 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:59.151842 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" event={"ID":"346d880e-ef45-406b-9a77-80760c094c37","Type":"ContainerStarted","Data":"70f185a1bd0ec33963b485ac7cae27c2a7286ebafee10cc645e7b3f9a28f5d17"} Nov 01 09:08:59.152005 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:59.151995 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:08:59.273348 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:08:59.273291 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" podStartSLOduration=2.2732763289999998 podStartE2EDuration="2.273276329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:08:59.271176518 +0000 UTC m=+2162.359513191" watchObservedRunningTime="2026-11-01 09:08:59.273276329 +0000 UTC m=+2162.361612998" Nov 01 09:09:02.643056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.643007 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj"] Nov 01 09:09:02.649388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.649360 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.701068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.701036 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj"] Nov 01 09:09:02.710978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.710931 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kzw\" (UniqueName: \"kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw\") pod \"payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.712133 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.712081 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.748602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.748567 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c3aaa7-dockercfg-hz8gr\"" Nov 01 09:09:02.750406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.748997 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c3aaa7\"" Nov 01 09:09:02.814024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.813990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.814218 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.814173 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kzw\" (UniqueName: \"kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw\") pod \"payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.814838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.814813 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.822982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.822947 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5"] Nov 01 09:09:02.827528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.827506 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:02.871783 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.871750 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kzw\" (UniqueName: \"kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw\") pod \"payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:02.879453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.879423 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5"] Nov 01 09:09:02.915259 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.915155 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7gw\" (UniqueName: \"kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw\") pod \"payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:02.915422 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.915408 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:02.966283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:02.966232 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:03.016693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.016650 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7gw\" (UniqueName: \"kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw\") pod \"payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:03.016897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.016878 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:03.017597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.017572 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:03.096385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.096342 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7gw\" (UniqueName: \"kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw\") pod \"payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:03.143033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.142517 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:03.350625 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.350586 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj"] Nov 01 09:09:03.357459 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:09:03.357429 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a24f164_c457_4d88_bfb6_0e2528630db4.slice/crio-bcd07be3be0d5e98a9e4f89898db9e05bf20cb120266f3e2001cfae89f85c07e WatchSource:0}: Error finding container bcd07be3be0d5e98a9e4f89898db9e05bf20cb120266f3e2001cfae89f85c07e: Status 404 returned error can't find the container with id bcd07be3be0d5e98a9e4f89898db9e05bf20cb120266f3e2001cfae89f85c07e Nov 01 09:09:03.512625 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.512593 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w"] Nov 01 09:09:03.517150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.517131 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.543244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.543205 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5"] Nov 01 09:09:03.552108 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:09:03.552070 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26846be6_9903_4d21_9e70_e32de7fbe366.slice/crio-980ef31a4f2930a52cb49723bd74052b393389418335919553b4291a0a7d5b89 WatchSource:0}: Error finding container 980ef31a4f2930a52cb49723bd74052b393389418335919553b4291a0a7d5b89: Status 404 returned error can't find the container with id 980ef31a4f2930a52cb49723bd74052b393389418335919553b4291a0a7d5b89 Nov 01 09:09:03.568297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.568254 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-616efa-serving-cert\"" Nov 01 09:09:03.572338 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.572311 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w"] Nov 01 09:09:03.631287 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.631255 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.631415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.631320 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wksm\" (UniqueName: \"kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.631503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.631480 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.631631 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.631521 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.732510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.732418 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.732510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.732466 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.733080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.732537 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.733080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.732568 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wksm\" (UniqueName: \"kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.733080 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:03.732672 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-serving-cert: secret "maas-api-e2e-praxis-a-616efa-serving-cert" not found Nov 01 09:09:03.733080 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:03.732784 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls podName:93935dae-3a7a-4ff3-a4fb-f193691b4147 nodeName:}" failed. No retries permitted until 2026-11-01 09:09:04.232764236 +0000 UTC m=+2167.321100886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls") pod "maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" (UID: "93935dae-3a7a-4ff3-a4fb-f193691b4147") : secret "maas-api-e2e-praxis-a-616efa-serving-cert" not found Nov 01 09:09:03.733277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.733257 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.735293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.735267 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:03.791339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:03.791297 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wksm\" (UniqueName: \"kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:04.189858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.189752 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" event={"ID":"8a24f164-c457-4d88-bfb6-0e2528630db4","Type":"ContainerStarted","Data":"533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6"} Nov 01 09:09:04.189858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.189818 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" event={"ID":"8a24f164-c457-4d88-bfb6-0e2528630db4","Type":"ContainerStarted","Data":"bcd07be3be0d5e98a9e4f89898db9e05bf20cb120266f3e2001cfae89f85c07e"} Nov 01 09:09:04.190124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.189957 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:04.191549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.191515 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" event={"ID":"26846be6-9903-4d21-9e70-e32de7fbe366","Type":"ContainerStarted","Data":"596796f7335ca4483347338df87bd4f8a9236575fbf0c6ae33ab727dd43e130d"} Nov 01 09:09:04.191677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.191572 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" event={"ID":"26846be6-9903-4d21-9e70-e32de7fbe366","Type":"ContainerStarted","Data":"980ef31a4f2930a52cb49723bd74052b393389418335919553b4291a0a7d5b89"} Nov 01 09:09:04.191677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.191602 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:04.244573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.244527 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:04.247201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.247174 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\") pod \"maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:04.299110 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.299033 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" podStartSLOduration=2.299015849 podStartE2EDuration="2.299015849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:04.298308893 +0000 UTC m=+2167.386645567" watchObservedRunningTime="2026-11-01 09:09:04.299015849 +0000 UTC m=+2167.387352520" Nov 01 09:09:04.424743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.412774 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" podStartSLOduration=2.408709787 podStartE2EDuration="2.408709787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:04.407267174 +0000 UTC m=+2167.495603847" watchObservedRunningTime="2026-11-01 09:09:04.408709787 +0000 UTC m=+2167.497046460" Nov 01 09:09:04.435164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.435115 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:04.817925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:04.817832 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w"] Nov 01 09:09:04.825322 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:09:04.825286 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93935dae_3a7a_4ff3_a4fb_f193691b4147.slice/crio-7abd7e3b70a47c09e7fc019fb2d3cd3d78cd1d613616d1bb2414959a3249c8ae WatchSource:0}: Error finding container 7abd7e3b70a47c09e7fc019fb2d3cd3d78cd1d613616d1bb2414959a3249c8ae: Status 404 returned error can't find the container with id 7abd7e3b70a47c09e7fc019fb2d3cd3d78cd1d613616d1bb2414959a3249c8ae Nov 01 09:09:05.199901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:05.199863 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" event={"ID":"93935dae-3a7a-4ff3-a4fb-f193691b4147","Type":"ContainerStarted","Data":"7abd7e3b70a47c09e7fc019fb2d3cd3d78cd1d613616d1bb2414959a3249c8ae"} Nov 01 09:09:06.208539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:06.208501 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" event={"ID":"93935dae-3a7a-4ff3-a4fb-f193691b4147","Type":"ContainerStarted","Data":"350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83"} Nov 01 09:09:06.208978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:06.208706 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:06.319457 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:06.319398 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" podStartSLOduration=2.395650549 podStartE2EDuration="3.319381324s" totalImagesPullingTime="923.730775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:06.318551926 +0000 UTC m=+2169.406888615" watchObservedRunningTime="2026-11-01 09:09:06.319381324 +0000 UTC m=+2169.407718021" Nov 01 09:09:09.154495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:09.154463 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:09:09.840491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:09.840457 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr"] Nov 01 09:09:09.939486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:09.939447 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r"] Nov 01 09:09:10.161454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:10.161380 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:09:12.151619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:12.151588 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sccmk_1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378/payload-processing/0.log" Nov 01 09:09:12.220228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:12.220196 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:09:12.383957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:12.383931 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sccmk_1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378/payload-processing/0.log" Nov 01 09:09:13.664628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.664590 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq"] Nov 01 09:09:13.669145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.669124 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.692128 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.692095 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.692320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.692175 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.692397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.692318 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kfp\" (UniqueName: \"kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.692487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.692471 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.724801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.724765 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq"] Nov 01 09:09:13.766760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.766702 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5fe64d-serving-cert\"" Nov 01 09:09:13.794261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.794216 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kfp\" (UniqueName: \"kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.794460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.794334 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.794460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.794409 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.794578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.794485 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.794643 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:13.794590 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-serving-cert: secret "maas-api-e2e-ipp-b-5fe64d-serving-cert" not found Nov 01 09:09:13.794700 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:13.794669 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls podName:15118450-ccc6-470e-ba4b-bed1fdffa8c2 nodeName:}" failed. No retries permitted until 2026-11-01 09:09:14.294646448 +0000 UTC m=+2177.382983102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls") pod "maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" (UID: "15118450-ccc6-470e-ba4b-bed1fdffa8c2") : secret "maas-api-e2e-ipp-b-5fe64d-serving-cert" not found Nov 01 09:09:13.795110 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.795085 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.797322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.797298 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:13.852258 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:13.852223 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kfp\" (UniqueName: \"kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:14.299866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:14.299831 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:14.302545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:14.302521 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:14.583521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:14.583419 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:14.841184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:14.841142 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" podUID="df642b66-bd84-4012-a8d8-3a3eabc6878e" containerName="payload-processing" containerID="cri-o://c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47" gracePeriod=25 Nov 01 09:09:14.939902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:14.939768 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" podUID="f76b035b-57e9-4f2b-bcc0-814437e63941" containerName="payload-pre-processing" containerID="cri-o://853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def" gracePeriod=25 Nov 01 09:09:14.965414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:14.965382 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq"] Nov 01 09:09:15.012822 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:09:15.012788 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15118450_ccc6_470e_ba4b_bed1fdffa8c2.slice/crio-d5d7fa9c87342467a5b6ea960d328ef6acffef5f1d18c086a9f7cd13af725f30 WatchSource:0}: Error finding container d5d7fa9c87342467a5b6ea960d328ef6acffef5f1d18c086a9f7cd13af725f30: Status 404 returned error can't find the container with id d5d7fa9c87342467a5b6ea960d328ef6acffef5f1d18c086a9f7cd13af725f30 Nov 01 09:09:15.119402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.119371 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:09:15.203058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.203031 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:09:15.203212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.203110 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:09:15.212029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.212003 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume\") pod \"df642b66-bd84-4012-a8d8-3a3eabc6878e\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " Nov 01 09:09:15.212203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.212110 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq\" (UniqueName: \"kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq\") pod \"df642b66-bd84-4012-a8d8-3a3eabc6878e\" (UID: \"df642b66-bd84-4012-a8d8-3a3eabc6878e\") " Nov 01 09:09:15.212435 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.212408 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume" pod "df642b66-bd84-4012-a8d8-3a3eabc6878e" (UID: "df642b66-bd84-4012-a8d8-3a3eabc6878e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:09:15.212577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.212517 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df642b66-bd84-4012-a8d8-3a3eabc6878e-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:15.215202 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.215175 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq" pod "df642b66-bd84-4012-a8d8-3a3eabc6878e" (UID: "df642b66-bd84-4012-a8d8-3a3eabc6878e"). InnerVolumeSpecName "kube-api-access-7s6jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:09:15.223918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.223897 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:09:15.287436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.287404 5710 generic.go:356] "Generic (PLEG): container finished" podID="df642b66-bd84-4012-a8d8-3a3eabc6878e" containerID="c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47" exitCode=0 Nov 01 09:09:15.287636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.287454 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" Nov 01 09:09:15.287636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.287494 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" event={"ID":"df642b66-bd84-4012-a8d8-3a3eabc6878e","Type":"ContainerDied","Data":"c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47"} Nov 01 09:09:15.287636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.287551 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr" event={"ID":"df642b66-bd84-4012-a8d8-3a3eabc6878e","Type":"ContainerDied","Data":"9d75431ff09126f2e919437bf993687501dd19937bc7d085164376d7096ca472"} Nov 01 09:09:15.287636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.287585 5710 scope.go:122] "RemoveContainer" containerID="c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47" Nov 01 09:09:15.288888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.288856 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" event={"ID":"15118450-ccc6-470e-ba4b-bed1fdffa8c2","Type":"ContainerStarted","Data":"d5d7fa9c87342467a5b6ea960d328ef6acffef5f1d18c086a9f7cd13af725f30"} Nov 01 09:09:15.290333 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.290304 5710 generic.go:356] "Generic (PLEG): container finished" podID="f76b035b-57e9-4f2b-bcc0-814437e63941" containerID="853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def" exitCode=0 Nov 01 09:09:15.290485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.290340 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" Nov 01 09:09:15.290485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.290388 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" event={"ID":"f76b035b-57e9-4f2b-bcc0-814437e63941","Type":"ContainerDied","Data":"853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def"} Nov 01 09:09:15.290485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.290440 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r" event={"ID":"f76b035b-57e9-4f2b-bcc0-814437e63941","Type":"ContainerDied","Data":"c06110835786aa19edeccbc82b7fffccbab9f02b017cf1d8716b3b10fc54b2aa"} Nov 01 09:09:15.298597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.298577 5710 scope.go:122] "RemoveContainer" containerID="c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47" Nov 01 09:09:15.298880 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:15.298860 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47\": container with ID starting with c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47 not found: ID does not exist" containerID="c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47" Nov 01 09:09:15.298976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.298891 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47"} err="failed to get container status \"c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47\": rpc error: code = NotFound desc = could not find container \"c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47\": container with ID starting with c247ef08e9e8a408048970fc2e79ffac83a9b8521f5c85397c3121a8c6328b47 not found: ID does not exist" Nov 01 09:09:15.298976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.298916 5710 scope.go:122] "RemoveContainer" containerID="853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def" Nov 01 09:09:15.308706 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.308689 5710 scope.go:122] "RemoveContainer" containerID="853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def" Nov 01 09:09:15.309013 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:15.308987 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def\": container with ID starting with 853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def not found: ID does not exist" containerID="853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def" Nov 01 09:09:15.309109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.309016 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def"} err="failed to get container status \"853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def\": rpc error: code = NotFound desc = could not find container \"853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def\": container with ID starting with 853ea7a926832c70e22d70df7a22ed1530fab053785669c425b86f6c9c768def not found: ID does not exist" Nov 01 09:09:15.313430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.313410 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume\") pod \"f76b035b-57e9-4f2b-bcc0-814437e63941\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " Nov 01 09:09:15.313513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.313459 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h\" (UniqueName: \"kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h\") pod \"f76b035b-57e9-4f2b-bcc0-814437e63941\" (UID: \"f76b035b-57e9-4f2b-bcc0-814437e63941\") " Nov 01 09:09:15.313782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.313765 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume" pod "f76b035b-57e9-4f2b-bcc0-814437e63941" (UID: "f76b035b-57e9-4f2b-bcc0-814437e63941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:09:15.314097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.314075 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76b035b-57e9-4f2b-bcc0-814437e63941-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:15.314199 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.314101 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s6jq\" (UniqueName: \"kubernetes.io/projected/df642b66-bd84-4012-a8d8-3a3eabc6878e-kube-api-access-7s6jq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:15.315658 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.315643 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h" pod "f76b035b-57e9-4f2b-bcc0-814437e63941" (UID: "f76b035b-57e9-4f2b-bcc0-814437e63941"). InnerVolumeSpecName "kube-api-access-5958h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:09:15.415194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.415113 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5958h\" (UniqueName: \"kubernetes.io/projected/f76b035b-57e9-4f2b-bcc0-814437e63941-kube-api-access-5958h\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:15.486631 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:15.486570 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:09:15.720705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.720616 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr"] Nov 01 09:09:15.724689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.724661 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-7fc88f8d59-c79pr"] Nov 01 09:09:15.829669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.829631 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r"] Nov 01 09:09:15.833654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:15.833622 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be7cc-85fbd8b599-9sj9r"] Nov 01 09:09:16.300272 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.300178 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" event={"ID":"15118450-ccc6-470e-ba4b-bed1fdffa8c2","Type":"ContainerStarted","Data":"e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc"} Nov 01 09:09:16.300766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.300287 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:16.413461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.413397 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" podStartSLOduration=2.523911214 podStartE2EDuration="3.413379725s" totalImagesPullingTime="889.468511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:16.411213272 +0000 UTC m=+2179.499549944" watchObservedRunningTime="2026-11-01 09:09:16.413379725 +0000 UTC m=+2179.501716396" Nov 01 09:09:16.728749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.728544 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq"] Nov 01 09:09:16.729887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.729627 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f76b035b-57e9-4f2b-bcc0-814437e63941" containerName="payload-pre-processing" Nov 01 09:09:16.729887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.729663 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76b035b-57e9-4f2b-bcc0-814437e63941" containerName="payload-pre-processing" Nov 01 09:09:16.737687 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.737644 5710 cpu_manager.go:414] "containerMap: removing container" podUID="df642b66-bd84-4012-a8d8-3a3eabc6878e" containerName="payload-processing" Nov 01 09:09:16.737687 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.737683 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df642b66-bd84-4012-a8d8-3a3eabc6878e" containerName="payload-processing" Nov 01 09:09:16.738074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.738030 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76b035b-57e9-4f2b-bcc0-814437e63941" containerName="payload-pre-processing" Nov 01 09:09:16.738074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.738067 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="df642b66-bd84-4012-a8d8-3a3eabc6878e" containerName="payload-processing" Nov 01 09:09:16.747512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.747472 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.779189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.779151 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq"] Nov 01 09:09:16.797284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.797242 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b893c8\"/\"default-dockercfg-hrfcf\"" Nov 01 09:09:16.821043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.821008 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b893c8\"/\"ipp-route-b893c8-kserve-self-signed-certs\"" Nov 01 09:09:16.821043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.821022 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b893c8\"/\"kube-root-ca.crt\"" Nov 01 09:09:16.821268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.821068 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b893c8\"/\"openshift-service-ca.crt\"" Nov 01 09:09:16.838897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.838859 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.839081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.838917 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.839081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.839024 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dnq\" (UniqueName: \"kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.839154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.839079 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.839194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.839150 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.839269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.839255 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940305 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940371 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dnq\" (UniqueName: \"kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940415 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940433 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940495 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940571 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940753 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.940989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940956 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.941119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.940991 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.942992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.942974 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.943278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.943261 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:16.998690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:16.998603 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dnq\" (UniqueName: \"kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq\") pod \"ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:17.059169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:17.059131 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df642b66-bd84-4012-a8d8-3a3eabc6878e" path="/var/lib/kubelet/pods/df642b66-bd84-4012-a8d8-3a3eabc6878e/volumes" Nov 01 09:09:17.059683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:17.059668 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76b035b-57e9-4f2b-bcc0-814437e63941" path="/var/lib/kubelet/pods/f76b035b-57e9-4f2b-bcc0-814437e63941/volumes" Nov 01 09:09:17.078676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:17.078649 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:17.460556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:17.460519 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq"] Nov 01 09:09:17.467231 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:09:17.467185 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf352dd2f_936b_400e_820b_7dcf9b4aaa4b.slice/crio-d01b5e0dcd7d1611e3d6d5fee05c68d0b51d46a744e5dde9e4a5f83ae40359da WatchSource:0}: Error finding container d01b5e0dcd7d1611e3d6d5fee05c68d0b51d46a744e5dde9e4a5f83ae40359da: Status 404 returned error can't find the container with id d01b5e0dcd7d1611e3d6d5fee05c68d0b51d46a744e5dde9e4a5f83ae40359da Nov 01 09:09:18.316601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:18.316564 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" event={"ID":"f352dd2f-936b-400e-820b-7dcf9b4aaa4b","Type":"ContainerStarted","Data":"ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e"} Nov 01 09:09:18.316835 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:18.316611 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" event={"ID":"f352dd2f-936b-400e-820b-7dcf9b4aaa4b","Type":"ContainerStarted","Data":"d01b5e0dcd7d1611e3d6d5fee05c68d0b51d46a744e5dde9e4a5f83ae40359da"} Nov 01 09:09:18.429607 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:18.429550 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" podStartSLOduration=2.4295340530000002 podStartE2EDuration="2.429534053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:09:18.42700222 +0000 UTC m=+2181.515338893" watchObservedRunningTime="2026-11-01 09:09:18.429534053 +0000 UTC m=+2181.517870724" Nov 01 09:09:19.079478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:19.079447 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:21.082791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:21.082710 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:09:22.315568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:22.315540 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:09:27.079685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:27.079648 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:27.093256 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:27.093223 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:27.386240 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:27.386210 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:09:45.479243 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:45.479192 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:09:46.949342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:46.949306 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:09:48.821304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:48.821263 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw"] Nov 01 09:09:48.821755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:48.821558 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" containerID="cri-o://5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa" gracePeriod=30 Nov 01 09:09:54.093446 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.093419 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 09:09:54.178468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178436 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178487 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178596 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178648 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178746 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178776 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178829 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.178868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178860 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.179062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178887 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:09:54.179062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.178911 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo\") pod \"41dde983-43b7-4c73-ae19-4955151dbf81\" (UID: \"41dde983-43b7-4c73-ae19-4955151dbf81\") " Nov 01 09:09:54.179062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179027 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:09:54.179297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179234 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:09:54.179508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179487 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:09:54.179599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179540 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:09:54.179665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179647 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.179742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179669 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41dde983-43b7-4c73-ae19-4955151dbf81-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.179742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.179684 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.181353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.181328 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:09:54.181834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.181805 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:09:54.184866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.184831 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:09:54.184963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.184860 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6" pod "41dde983-43b7-4c73-ae19-4955151dbf81" (UID: "41dde983-43b7-4c73-ae19-4955151dbf81"). InnerVolumeSpecName "kube-api-access-mnbs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:09:54.280900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.280857 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.280900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.280888 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.280900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.280898 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41dde983-43b7-4c73-ae19-4955151dbf81-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.280900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.280907 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41dde983-43b7-4c73-ae19-4955151dbf81-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.281286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.280916 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.281286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.280925 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbs6\" (UniqueName: \"kubernetes.io/projected/41dde983-43b7-4c73-ae19-4955151dbf81-kube-api-access-mnbs6\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:09:54.559624 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.559532 5710 generic.go:356] "Generic (PLEG): container finished" podID="41dde983-43b7-4c73-ae19-4955151dbf81" containerID="5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa" exitCode=0 Nov 01 09:09:54.559867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.559656 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" Nov 01 09:09:54.559867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.559677 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" event={"ID":"41dde983-43b7-4c73-ae19-4955151dbf81","Type":"ContainerDied","Data":"5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa"} Nov 01 09:09:54.559867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.559810 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw" event={"ID":"41dde983-43b7-4c73-ae19-4955151dbf81","Type":"ContainerDied","Data":"0d7dffcd19018429806f7026dc200f4025c72f20a8e5f87e01ebdc433fecb348"} Nov 01 09:09:54.559867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.559843 5710 scope.go:122] "RemoveContainer" containerID="5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa" Nov 01 09:09:54.571895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.571873 5710 scope.go:122] "RemoveContainer" containerID="32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72" Nov 01 09:09:54.583761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.583728 5710 scope.go:122] "RemoveContainer" containerID="5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa" Nov 01 09:09:54.584217 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:54.584178 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa\": container with ID starting with 5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa not found: ID does not exist" containerID="5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa" Nov 01 09:09:54.584328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.584217 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa"} err="failed to get container status \"5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa\": rpc error: code = NotFound desc = could not find container \"5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa\": container with ID starting with 5e5789fea72a109bc3894ae44928c72e2c3832ee683ae4e39543b767645a56fa not found: ID does not exist" Nov 01 09:09:54.584328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.584245 5710 scope.go:122] "RemoveContainer" containerID="32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72" Nov 01 09:09:54.584663 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:09:54.584639 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72\": container with ID starting with 32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72 not found: ID does not exist" containerID="32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72" Nov 01 09:09:54.584896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.584675 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72"} err="failed to get container status \"32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72\": rpc error: code = NotFound desc = could not find container \"32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72\": container with ID starting with 32019dcb946915b63925aa6737f5ca27673cdc7209b865b00296d351b40b9b72 not found: ID does not exist" Nov 01 09:09:54.719479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.719432 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw"] Nov 01 09:09:54.723234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:54.723192 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-6ldrw"] Nov 01 09:09:55.059129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:09:55.059094 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" path="/var/lib/kubelet/pods/41dde983-43b7-4c73-ae19-4955151dbf81/volumes" Nov 01 09:10:15.480998 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:15.480954 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:10:22.066860 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.066824 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf"] Nov 01 09:10:22.067495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.067480 5710 cpu_manager.go:414] "containerMap: removing container" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" Nov 01 09:10:22.067552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.067496 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" Nov 01 09:10:22.067552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.067517 5710 cpu_manager.go:414] "containerMap: removing container" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" Nov 01 09:10:22.067552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.067523 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" Nov 01 09:10:22.067650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.067616 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" Nov 01 09:10:22.071241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.071216 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.169103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.169057 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4be7cc\"" Nov 01 09:10:22.169103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.169080 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4be7cc-dockercfg-8fd6w\"" Nov 01 09:10:22.171826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.171796 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf"] Nov 01 09:10:22.172394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.172369 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcpl\" (UniqueName: \"kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl\") pod \"payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.172528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.172513 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.274369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.274329 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcpl\" (UniqueName: \"kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl\") pod \"payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.274591 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.274441 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.275281 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.275254 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.336527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.336433 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcpl\" (UniqueName: \"kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl\") pod \"payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.386600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.386560 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:22.782712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:22.782670 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf"] Nov 01 09:10:22.790098 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:10:22.790064 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da49772_4bf4_42eb_9a19_8b91024cc1e2.slice/crio-e5f71a9fcb96fac1cc9322062955c4fe53b93f24921b94de49d3e2dc2dfa071d WatchSource:0}: Error finding container e5f71a9fcb96fac1cc9322062955c4fe53b93f24921b94de49d3e2dc2dfa071d: Status 404 returned error can't find the container with id e5f71a9fcb96fac1cc9322062955c4fe53b93f24921b94de49d3e2dc2dfa071d Nov 01 09:10:23.794327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:23.794289 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" event={"ID":"9da49772-4bf4-42eb-9a19-8b91024cc1e2","Type":"ContainerStarted","Data":"122037902d7b4e0acdf707788514c36a291694dda56f53df833e28bbdccb69c9"} Nov 01 09:10:23.794866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:23.794351 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" event={"ID":"9da49772-4bf4-42eb-9a19-8b91024cc1e2","Type":"ContainerStarted","Data":"e5f71a9fcb96fac1cc9322062955c4fe53b93f24921b94de49d3e2dc2dfa071d"} Nov 01 09:10:23.794866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:23.794391 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:23.912274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:23.912210 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" podStartSLOduration=1.912192792 podStartE2EDuration="1.912192792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:10:23.910698224 +0000 UTC m=+2246.999034900" watchObservedRunningTime="2026-11-01 09:10:23.912192792 +0000 UTC m=+2247.000529540" Nov 01 09:10:33.352795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.352748 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw"] Nov 01 09:10:33.353548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.353532 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="41dde983-43b7-4c73-ae19-4955151dbf81" containerName="istio-proxy" Nov 01 09:10:33.357860 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.357834 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.407045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.406996 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9f4e9b-openshift-default-dockercfg-4rl7t\"" Nov 01 09:10:33.415975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.415918 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416123 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49vr\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416156 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416247 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416268 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416370 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416432 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416514 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.416568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.416544 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.456965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.456926 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw"] Nov 01 09:10:33.528644 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528601 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.528915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528655 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.528915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528696 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.528915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528769 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.528915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528826 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.528915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528853 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.529211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.528945 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.529211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.529146 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49vr\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.529211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.529181 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.529701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.529665 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.532811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.532783 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.533616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.533593 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.534072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.534052 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.534842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.534822 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.538051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.537645 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.541224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.541176 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.615961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.615873 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49vr\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.617240 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.617208 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token\") pod \"e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:33.672962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:33.672929 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:34.137209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.137176 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw"] Nov 01 09:10:34.144573 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:10:34.144540 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dbd1a1_748a_4956_86db_8d248a00053f.slice/crio-61cd606dd442b7dfa6431c74b0c0fe2ec06e77f0a50176cb976c4c09673df77d WatchSource:0}: Error finding container 61cd606dd442b7dfa6431c74b0c0fe2ec06e77f0a50176cb976c4c09673df77d: Status 404 returned error can't find the container with id 61cd606dd442b7dfa6431c74b0c0fe2ec06e77f0a50176cb976c4c09673df77d Nov 01 09:10:34.147596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.147561 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:10:34.147709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.147640 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:10:34.147709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.147670 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:10:34.803821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.803786 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:10:34.866220 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.866177 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" event={"ID":"e1dbd1a1-748a-4956-86db-8d248a00053f","Type":"ContainerStarted","Data":"e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603"} Nov 01 09:10:34.866456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:34.866254 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" event={"ID":"e1dbd1a1-748a-4956-86db-8d248a00053f","Type":"ContainerStarted","Data":"61cd606dd442b7dfa6431c74b0c0fe2ec06e77f0a50176cb976c4c09673df77d"} Nov 01 09:10:35.029949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:35.029882 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podStartSLOduration=2.029860419 podStartE2EDuration="2.029860419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:10:35.027589926 +0000 UTC m=+2258.115926593" watchObservedRunningTime="2026-11-01 09:10:35.029860419 +0000 UTC m=+2258.118197093" Nov 01 09:10:35.673234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:35.673186 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:35.679838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:35.679810 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:35.873278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:35.873244 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:35.874446 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:35.874424 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:10:37.206037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:37.206000 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:10:42.658856 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:42.658821 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt"] Nov 01 09:10:43.370853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.370816 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx"] Nov 01 09:10:43.371078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.371038 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" podUID="9ea9f9c4-10f7-44c5-915e-c397521824ea" containerName="maas-api" containerID="cri-o://8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9" gracePeriod=30 Nov 01 09:10:43.749243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.749202 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:10:43.751090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.751060 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs\") pod \"9ea9f9c4-10f7-44c5-915e-c397521824ea\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " Nov 01 09:10:43.751284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.751108 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\") pod \"9ea9f9c4-10f7-44c5-915e-c397521824ea\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " Nov 01 09:10:43.751284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.751134 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z\" (UniqueName: \"kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z\") pod \"9ea9f9c4-10f7-44c5-915e-c397521824ea\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " Nov 01 09:10:43.751284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.751164 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca\") pod \"9ea9f9c4-10f7-44c5-915e-c397521824ea\" (UID: \"9ea9f9c4-10f7-44c5-915e-c397521824ea\") " Nov 01 09:10:43.751659 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.751624 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca" pod "9ea9f9c4-10f7-44c5-915e-c397521824ea" (UID: "9ea9f9c4-10f7-44c5-915e-c397521824ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:10:43.753653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.753618 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls" pod "9ea9f9c4-10f7-44c5-915e-c397521824ea" (UID: "9ea9f9c4-10f7-44c5-915e-c397521824ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:10:43.753987 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.753964 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs" pod "9ea9f9c4-10f7-44c5-915e-c397521824ea" (UID: "9ea9f9c4-10f7-44c5-915e-c397521824ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:10:43.754089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.754008 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z" pod "9ea9f9c4-10f7-44c5-915e-c397521824ea" (UID: "9ea9f9c4-10f7-44c5-915e-c397521824ea"). InnerVolumeSpecName "kube-api-access-ljm6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:10:43.852513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.852469 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:43.852513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.852507 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea9f9c4-10f7-44c5-915e-c397521824ea-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:43.852513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.852519 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljm6z\" (UniqueName: \"kubernetes.io/projected/9ea9f9c4-10f7-44c5-915e-c397521824ea-kube-api-access-ljm6z\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:43.852818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.852531 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea9f9c4-10f7-44c5-915e-c397521824ea-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:43.921123 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.921075 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7"] Nov 01 09:10:43.921369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.921337 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" podUID="10c9ac6c-1566-4f9d-8478-46fc13d615fa" containerName="payload-processing" containerID="cri-o://1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b" gracePeriod=30 Nov 01 09:10:43.961563 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.961524 5710 generic.go:356] "Generic (PLEG): container finished" podID="9ea9f9c4-10f7-44c5-915e-c397521824ea" containerID="8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9" exitCode=0 Nov 01 09:10:43.961777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.961577 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" Nov 01 09:10:43.961777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.961606 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" event={"ID":"9ea9f9c4-10f7-44c5-915e-c397521824ea","Type":"ContainerDied","Data":"8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9"} Nov 01 09:10:43.961777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.961666 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx" event={"ID":"9ea9f9c4-10f7-44c5-915e-c397521824ea","Type":"ContainerDied","Data":"040e5800b894dae911786a1ee344cb869eb1d977b20aa39b438ee519c23283fb"} Nov 01 09:10:43.961777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.961690 5710 scope.go:122] "RemoveContainer" containerID="8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9" Nov 01 09:10:43.976372 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.976283 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94"] Nov 01 09:10:43.976587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:43.976560 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" podUID="03a163c1-ee6e-42b2-910e-81bf40f30097" containerName="payload-pre-processing" containerID="cri-o://b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913" gracePeriod=30 Nov 01 09:10:44.065485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.064932 5710 scope.go:122] "RemoveContainer" containerID="8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9" Nov 01 09:10:44.065485 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:44.065416 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9\": container with ID starting with 8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9 not found: ID does not exist" containerID="8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9" Nov 01 09:10:44.065485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.065452 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9"} err="failed to get container status \"8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9\": rpc error: code = NotFound desc = could not find container \"8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9\": container with ID starting with 8fc255360c3a7d302b9e0bb3a6b3c155be3f3c5d086273ff03db0798a7de59e9 not found: ID does not exist" Nov 01 09:10:44.265777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.254155 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:10:44.265777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.257571 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x\") pod \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " Nov 01 09:10:44.265777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.257782 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume\") pod \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\" (UID: \"10c9ac6c-1566-4f9d-8478-46fc13d615fa\") " Nov 01 09:10:44.265777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.258120 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume" pod "10c9ac6c-1566-4f9d-8478-46fc13d615fa" (UID: "10c9ac6c-1566-4f9d-8478-46fc13d615fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:10:44.265777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.258323 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c9ac6c-1566-4f9d-8478-46fc13d615fa-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:44.265777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.262222 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x" pod "10c9ac6c-1566-4f9d-8478-46fc13d615fa" (UID: "10c9ac6c-1566-4f9d-8478-46fc13d615fa"). InnerVolumeSpecName "kube-api-access-8m57x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:10:44.308306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.308270 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:10:44.323605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.323567 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx"] Nov 01 09:10:44.328418 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.328377 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-095909-5898fd89fd-485zx"] Nov 01 09:10:44.359608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.359569 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume\") pod \"03a163c1-ee6e-42b2-910e-81bf40f30097\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " Nov 01 09:10:44.359821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.359808 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc\" (UniqueName: \"kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc\") pod \"03a163c1-ee6e-42b2-910e-81bf40f30097\" (UID: \"03a163c1-ee6e-42b2-910e-81bf40f30097\") " Nov 01 09:10:44.360024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.359998 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume" pod "03a163c1-ee6e-42b2-910e-81bf40f30097" (UID: "03a163c1-ee6e-42b2-910e-81bf40f30097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:10:44.360231 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.360211 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a163c1-ee6e-42b2-910e-81bf40f30097-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:44.360231 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.360229 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/10c9ac6c-1566-4f9d-8478-46fc13d615fa-kube-api-access-8m57x\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:44.362366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.362332 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc" pod "03a163c1-ee6e-42b2-910e-81bf40f30097" (UID: "03a163c1-ee6e-42b2-910e-81bf40f30097"). InnerVolumeSpecName "kube-api-access-wrvhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:10:44.461867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.461818 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrvhc\" (UniqueName: \"kubernetes.io/projected/03a163c1-ee6e-42b2-910e-81bf40f30097-kube-api-access-wrvhc\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:44.971221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.971175 5710 generic.go:356] "Generic (PLEG): container finished" podID="03a163c1-ee6e-42b2-910e-81bf40f30097" containerID="b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913" exitCode=0 Nov 01 09:10:44.971775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.971264 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" Nov 01 09:10:44.971775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.971265 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" event={"ID":"03a163c1-ee6e-42b2-910e-81bf40f30097","Type":"ContainerDied","Data":"b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913"} Nov 01 09:10:44.971775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.971372 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94" event={"ID":"03a163c1-ee6e-42b2-910e-81bf40f30097","Type":"ContainerDied","Data":"fc82054d268a42bf66053e8d41ef3a322cfa2ada5bace9a2618834f3394bf887"} Nov 01 09:10:44.971775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.971391 5710 scope.go:122] "RemoveContainer" containerID="b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913" Nov 01 09:10:44.972948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.972925 5710 generic.go:356] "Generic (PLEG): container finished" podID="10c9ac6c-1566-4f9d-8478-46fc13d615fa" containerID="1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b" exitCode=0 Nov 01 09:10:44.973065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.972959 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" event={"ID":"10c9ac6c-1566-4f9d-8478-46fc13d615fa","Type":"ContainerDied","Data":"1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b"} Nov 01 09:10:44.973129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.972992 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" Nov 01 09:10:44.973196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.973178 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7" event={"ID":"10c9ac6c-1566-4f9d-8478-46fc13d615fa","Type":"ContainerDied","Data":"114fea0ee9587238be7b9e7e38406f26cad8bf5f058044a2e1297f6dec5f1fb3"} Nov 01 09:10:44.982546 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.982522 5710 scope.go:122] "RemoveContainer" containerID="b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913" Nov 01 09:10:44.982934 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:44.982906 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913\": container with ID starting with b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913 not found: ID does not exist" containerID="b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913" Nov 01 09:10:44.983007 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.982945 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913"} err="failed to get container status \"b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913\": rpc error: code = NotFound desc = could not find container \"b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913\": container with ID starting with b760681072c47de163873eab7fca5cf925e604ba59211e9a05706c2e40eda913 not found: ID does not exist" Nov 01 09:10:44.983007 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.982965 5710 scope.go:122] "RemoveContainer" containerID="1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b" Nov 01 09:10:44.994431 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.994408 5710 scope.go:122] "RemoveContainer" containerID="1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b" Nov 01 09:10:44.994846 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:44.994817 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b\": container with ID starting with 1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b not found: ID does not exist" containerID="1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b" Nov 01 09:10:44.994967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:44.994857 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b"} err="failed to get container status \"1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b\": rpc error: code = NotFound desc = could not find container \"1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b\": container with ID starting with 1a3d3186e39469206f4755112e94885b34e1a8fbede79c2fb88ac8a3f2440d8b not found: ID does not exist" Nov 01 09:10:45.062087 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:45.062054 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea9f9c4-10f7-44c5-915e-c397521824ea" path="/var/lib/kubelet/pods/9ea9f9c4-10f7-44c5-915e-c397521824ea/volumes" Nov 01 09:10:45.125351 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:45.125308 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7"] Nov 01 09:10:45.129311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:45.129274 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-095909-5c84d4558-7rrx7"] Nov 01 09:10:45.231760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:45.231665 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94"] Nov 01 09:10:45.235469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:45.235438 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-095909-584bd4d7c9-h4d94"] Nov 01 09:10:45.483357 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:45.483249 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:10:47.058509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:47.058477 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a163c1-ee6e-42b2-910e-81bf40f30097" path="/var/lib/kubelet/pods/03a163c1-ee6e-42b2-910e-81bf40f30097/volumes" Nov 01 09:10:47.058962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:47.058945 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c9ac6c-1566-4f9d-8478-46fc13d615fa" path="/var/lib/kubelet/pods/10c9ac6c-1566-4f9d-8478-46fc13d615fa/volumes" Nov 01 09:10:47.620468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:47.620417 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr"] Nov 01 09:10:47.620708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:47.620663 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" containerID="cri-o://a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920" gracePeriod=30 Nov 01 09:10:47.659524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:47.659483 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" podUID="b4628121-3162-4d72-a705-1f815ca00a56" containerName="payload-processing" containerID="cri-o://8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd" gracePeriod=25 Nov 01 09:10:47.962572 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:47.962539 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:10:48.005698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.005659 5710 generic.go:356] "Generic (PLEG): container finished" podID="b4628121-3162-4d72-a705-1f815ca00a56" containerID="8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd" exitCode=0 Nov 01 09:10:48.005902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.005747 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" Nov 01 09:10:48.005902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.005753 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" event={"ID":"b4628121-3162-4d72-a705-1f815ca00a56","Type":"ContainerDied","Data":"8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd"} Nov 01 09:10:48.005902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.005823 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt" event={"ID":"b4628121-3162-4d72-a705-1f815ca00a56","Type":"ContainerDied","Data":"277ebb382f7a484e101b761397c732f9bd2fa8df3fd326ebd5f56cb72c7c6ad9"} Nov 01 09:10:48.005902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.005852 5710 scope.go:122] "RemoveContainer" containerID="8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd" Nov 01 09:10:48.016449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.016427 5710 scope.go:122] "RemoveContainer" containerID="8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd" Nov 01 09:10:48.016808 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:48.016780 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd\": container with ID starting with 8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd not found: ID does not exist" containerID="8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd" Nov 01 09:10:48.016863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.016819 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd"} err="failed to get container status \"8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd\": rpc error: code = NotFound desc = could not find container \"8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd\": container with ID starting with 8ffbc51bf63c615e4d7cbb7d69401aaf1fcae71b9bc275cfd6ebc7564851adbd not found: ID does not exist" Nov 01 09:10:48.019801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.019775 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume\") pod \"b4628121-3162-4d72-a705-1f815ca00a56\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " Nov 01 09:10:48.019979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.019958 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml\" (UniqueName: \"kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml\") pod \"b4628121-3162-4d72-a705-1f815ca00a56\" (UID: \"b4628121-3162-4d72-a705-1f815ca00a56\") " Nov 01 09:10:48.020204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.020176 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume" pod "b4628121-3162-4d72-a705-1f815ca00a56" (UID: "b4628121-3162-4d72-a705-1f815ca00a56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:10:48.020522 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.020500 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4628121-3162-4d72-a705-1f815ca00a56-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:48.022361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.022330 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml" pod "b4628121-3162-4d72-a705-1f815ca00a56" (UID: "b4628121-3162-4d72-a705-1f815ca00a56"). InnerVolumeSpecName "kube-api-access-4l8ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:10:48.121356 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.121318 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8ml\" (UniqueName: \"kubernetes.io/projected/b4628121-3162-4d72-a705-1f815ca00a56-kube-api-access-4l8ml\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:48.420784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.420735 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt"] Nov 01 09:10:48.424341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:48.424309 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5fe64d-66d55bf8d7-nfxwt"] Nov 01 09:10:49.014221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.014179 5710 generic.go:356] "Generic (PLEG): container finished" podID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerID="e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202" exitCode=0 Nov 01 09:10:49.014434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.014251 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" event={"ID":"63fd7f11-48e2-4703-af42-22d29b0cd125","Type":"ContainerDied","Data":"e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202"} Nov 01 09:10:49.014776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.014753 5710 scope.go:122] "RemoveContainer" containerID="e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202" Nov 01 09:10:49.016472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.016443 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:10:49.016542 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.016518 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:10:49.016582 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.016552 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:10:49.059941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:49.059906 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4628121-3162-4d72-a705-1f815ca00a56" path="/var/lib/kubelet/pods/b4628121-3162-4d72-a705-1f815ca00a56/volumes" Nov 01 09:10:50.024731 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:50.024684 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" event={"ID":"63fd7f11-48e2-4703-af42-22d29b0cd125","Type":"ContainerStarted","Data":"451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842"} Nov 01 09:10:50.244999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:50.244957 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:10:50.245202 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:50.245155 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 09:10:50.245250 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:50.245221 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 09:10:51.245084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:51.245047 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 09:10:51.245599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:51.245105 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 09:10:52.245392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:52.245346 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 09:10:52.245918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:52.245420 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 09:10:52.948227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:52.948198 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:10:53.045507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.045473 5710 generic.go:356] "Generic (PLEG): container finished" podID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerID="a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920" exitCode=0 Nov 01 09:10:53.045693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.045533 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" Nov 01 09:10:53.045693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.045559 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" event={"ID":"bd31e0d3-924d-400c-bb26-fccdcfd388b0","Type":"ContainerDied","Data":"a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920"} Nov 01 09:10:53.045693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.045619 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr" event={"ID":"bd31e0d3-924d-400c-bb26-fccdcfd388b0","Type":"ContainerDied","Data":"cbfd28849031983a1befc266b9e914359ff95ac65cf135c3bb645d16ccf9ac4b"} Nov 01 09:10:53.045693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.045646 5710 scope.go:122] "RemoveContainer" containerID="a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920" Nov 01 09:10:53.057828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.057805 5710 scope.go:122] "RemoveContainer" containerID="02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d" Nov 01 09:10:53.067795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.067774 5710 scope.go:122] "RemoveContainer" containerID="a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920" Nov 01 09:10:53.068129 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:53.068108 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920\": container with ID starting with a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920 not found: ID does not exist" containerID="a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920" Nov 01 09:10:53.068175 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.068145 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920"} err="failed to get container status \"a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920\": rpc error: code = NotFound desc = could not find container \"a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920\": container with ID starting with a0ac0dc072118c0ad5ce159ed306c21d52eecaf7df2b37fea0ac629efd8f9920 not found: ID does not exist" Nov 01 09:10:53.068175 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.068165 5710 scope.go:122] "RemoveContainer" containerID="02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d" Nov 01 09:10:53.068408 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:10:53.068390 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d\": container with ID starting with 02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d not found: ID does not exist" containerID="02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d" Nov 01 09:10:53.068447 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.068412 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d"} err="failed to get container status \"02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d\": rpc error: code = NotFound desc = could not find container \"02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d\": container with ID starting with 02e7ef50572f3065695bd5b5371d79bd71743021bfbfe93732ba273cd51ac97d not found: ID does not exist" Nov 01 09:10:53.107251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107192 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107282 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107315 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107391 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107474 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107506 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107536 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:10:53.107647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107611 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107659 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107693 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data\") pod \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\" (UID: \"bd31e0d3-924d-400c-bb26-fccdcfd388b0\") " Nov 01 09:10:53.107884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107743 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:10:53.107884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.107865 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:10:53.108120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108097 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:10:53.108247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108217 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:10:53.108575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108455 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.108575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108482 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.108575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108499 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.108575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108513 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.108575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.108528 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.109973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.109939 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:10:53.110094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.109994 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:10:53.110094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.110031 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:10:53.110179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.110155 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d" pod "bd31e0d3-924d-400c-bb26-fccdcfd388b0" (UID: "bd31e0d3-924d-400c-bb26-fccdcfd388b0"). InnerVolumeSpecName "kube-api-access-hk68d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:10:53.209914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.209813 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.210078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.209899 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk68d\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-kube-api-access-hk68d\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.210078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.209945 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.210078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.209960 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd31e0d3-924d-400c-bb26-fccdcfd388b0-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:10:53.245093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.245048 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 09:10:53.245281 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.245128 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 09:10:53.469467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.469372 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr"] Nov 01 09:10:53.473229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:53.473194 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-095909-openshift-default-fbb4b66f5-rtzkr"] Nov 01 09:10:54.245847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:54.245800 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 09:10:54.246061 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:54.245864 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 09:10:55.059443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:55.059404 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" path="/var/lib/kubelet/pods/bd31e0d3-924d-400c-bb26-fccdcfd388b0/volumes" Nov 01 09:10:55.245866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:55.245817 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 09:10:55.246062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:55.245900 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 09:10:56.250311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:56.250273 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:10:56.250859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:56.250675 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:10:56.251331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:56.251311 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:10:59.829042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:10:59.829010 5710 scope.go:122] "RemoveContainer" containerID="915c38cc2a470e6a9ef12cdc47a5094dda198818962618168700ca44bde35bbd" Nov 01 09:11:15.485386 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:15.485346 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:11:19.125829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.125789 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg"] Nov 01 09:11:19.126430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.126010 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" podUID="517fc65a-4127-4b76-946b-2b46862047d4" containerName="maas-api" containerID="cri-o://2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210" gracePeriod=30 Nov 01 09:11:19.420515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.420478 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:11:19.542976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.542930 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz\" (UniqueName: \"kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz\") pod \"517fc65a-4127-4b76-946b-2b46862047d4\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " Nov 01 09:11:19.544025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.543704 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs\") pod \"517fc65a-4127-4b76-946b-2b46862047d4\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " Nov 01 09:11:19.545006 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.544985 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\") pod \"517fc65a-4127-4b76-946b-2b46862047d4\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " Nov 01 09:11:19.545517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.545495 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca\") pod \"517fc65a-4127-4b76-946b-2b46862047d4\" (UID: \"517fc65a-4127-4b76-946b-2b46862047d4\") " Nov 01 09:11:19.546444 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.546314 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca" pod "517fc65a-4127-4b76-946b-2b46862047d4" (UID: "517fc65a-4127-4b76-946b-2b46862047d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:11:19.547313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.547291 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517fc65a-4127-4b76-946b-2b46862047d4-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:19.547539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.547520 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs" pod "517fc65a-4127-4b76-946b-2b46862047d4" (UID: "517fc65a-4127-4b76-946b-2b46862047d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:11:19.554306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.554259 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz" pod "517fc65a-4127-4b76-946b-2b46862047d4" (UID: "517fc65a-4127-4b76-946b-2b46862047d4"). InnerVolumeSpecName "kube-api-access-x49cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:11:19.556863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.556810 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls" pod "517fc65a-4127-4b76-946b-2b46862047d4" (UID: "517fc65a-4127-4b76-946b-2b46862047d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:11:19.579837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.579797 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68"] Nov 01 09:11:19.580152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.580099 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" podUID="346d880e-ef45-406b-9a77-80760c094c37" containerName="payload-processing" containerID="cri-o://de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09" gracePeriod=30 Nov 01 09:11:19.635545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.635504 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv"] Nov 01 09:11:19.635752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.635705 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" podUID="f15350a6-9f01-49e9-a850-c210c8daaa2d" containerName="payload-pre-processing" containerID="cri-o://e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca" gracePeriod=30 Nov 01 09:11:19.649034 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.648947 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x49cz\" (UniqueName: \"kubernetes.io/projected/517fc65a-4127-4b76-946b-2b46862047d4-kube-api-access-x49cz\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:19.649034 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.648984 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:19.649034 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.648994 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517fc65a-4127-4b76-946b-2b46862047d4-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:19.928352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.928309 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:11:19.959953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:19.959918 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:11:20.057315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.057277 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8\" (UniqueName: \"kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8\") pod \"f15350a6-9f01-49e9-a850-c210c8daaa2d\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " Nov 01 09:11:20.057528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.057360 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume\") pod \"f15350a6-9f01-49e9-a850-c210c8daaa2d\" (UID: \"f15350a6-9f01-49e9-a850-c210c8daaa2d\") " Nov 01 09:11:20.057528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.057452 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume\") pod \"346d880e-ef45-406b-9a77-80760c094c37\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " Nov 01 09:11:20.057768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.057597 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj\" (UniqueName: \"kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj\") pod \"346d880e-ef45-406b-9a77-80760c094c37\" (UID: \"346d880e-ef45-406b-9a77-80760c094c37\") " Nov 01 09:11:20.057897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.057860 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume" pod "f15350a6-9f01-49e9-a850-c210c8daaa2d" (UID: "f15350a6-9f01-49e9-a850-c210c8daaa2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:11:20.057965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.057895 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume" pod "346d880e-ef45-406b-9a77-80760c094c37" (UID: "346d880e-ef45-406b-9a77-80760c094c37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:11:20.058201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.058182 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15350a6-9f01-49e9-a850-c210c8daaa2d-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:20.058278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.058208 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346d880e-ef45-406b-9a77-80760c094c37-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:20.060020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.059991 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj" pod "346d880e-ef45-406b-9a77-80760c094c37" (UID: "346d880e-ef45-406b-9a77-80760c094c37"). InnerVolumeSpecName "kube-api-access-6jzhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:11:20.060341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.060311 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8" pod "f15350a6-9f01-49e9-a850-c210c8daaa2d" (UID: "f15350a6-9f01-49e9-a850-c210c8daaa2d"). InnerVolumeSpecName "kube-api-access-5prj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:11:20.161890 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.161847 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzhj\" (UniqueName: \"kubernetes.io/projected/346d880e-ef45-406b-9a77-80760c094c37-kube-api-access-6jzhj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:20.161890 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.161888 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prj8\" (UniqueName: \"kubernetes.io/projected/f15350a6-9f01-49e9-a850-c210c8daaa2d-kube-api-access-5prj8\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:20.226636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.226548 5710 generic.go:356] "Generic (PLEG): container finished" podID="f15350a6-9f01-49e9-a850-c210c8daaa2d" containerID="e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca" exitCode=0 Nov 01 09:11:20.226636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.226593 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" Nov 01 09:11:20.226919 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.226637 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" event={"ID":"f15350a6-9f01-49e9-a850-c210c8daaa2d","Type":"ContainerDied","Data":"e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca"} Nov 01 09:11:20.226919 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.226688 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv" event={"ID":"f15350a6-9f01-49e9-a850-c210c8daaa2d","Type":"ContainerDied","Data":"c936a9904f843f18912e31b1c4ce486acedcb5e23416c9cd22c77570e79881e9"} Nov 01 09:11:20.226919 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.226755 5710 scope.go:122] "RemoveContainer" containerID="e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca" Nov 01 09:11:20.228204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.228186 5710 generic.go:356] "Generic (PLEG): container finished" podID="346d880e-ef45-406b-9a77-80760c094c37" containerID="de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09" exitCode=0 Nov 01 09:11:20.228303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.228246 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" Nov 01 09:11:20.228303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.228266 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" event={"ID":"346d880e-ef45-406b-9a77-80760c094c37","Type":"ContainerDied","Data":"de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09"} Nov 01 09:11:20.228413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.228315 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68" event={"ID":"346d880e-ef45-406b-9a77-80760c094c37","Type":"ContainerDied","Data":"70f185a1bd0ec33963b485ac7cae27c2a7286ebafee10cc645e7b3f9a28f5d17"} Nov 01 09:11:20.230060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.230034 5710 generic.go:356] "Generic (PLEG): container finished" podID="517fc65a-4127-4b76-946b-2b46862047d4" containerID="2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210" exitCode=0 Nov 01 09:11:20.230193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.230092 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" Nov 01 09:11:20.230193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.230102 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" event={"ID":"517fc65a-4127-4b76-946b-2b46862047d4","Type":"ContainerDied","Data":"2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210"} Nov 01 09:11:20.230193 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.230145 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg" event={"ID":"517fc65a-4127-4b76-946b-2b46862047d4","Type":"ContainerDied","Data":"0e56d622a0e42cb1d11185c28ad8fe0bd02c721068a1a6d9870197c33284a51a"} Nov 01 09:11:20.238273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.237993 5710 scope.go:122] "RemoveContainer" containerID="e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca" Nov 01 09:11:20.249646 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:20.238370 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca\": container with ID starting with e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca not found: ID does not exist" containerID="e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca" Nov 01 09:11:20.249646 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.238878 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca"} err="failed to get container status \"e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca\": rpc error: code = NotFound desc = could not find container \"e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca\": container with ID starting with e84ada79b77aa308dcc9343edcc2f2664aca71d9828a93f625a0045e46fa78ca not found: ID does not exist" Nov 01 09:11:20.249646 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.238917 5710 scope.go:122] "RemoveContainer" containerID="de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09" Nov 01 09:11:20.258758 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.258734 5710 scope.go:122] "RemoveContainer" containerID="de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09" Nov 01 09:11:20.259214 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:20.259177 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09\": container with ID starting with de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09 not found: ID does not exist" containerID="de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09" Nov 01 09:11:20.259282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.259233 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09"} err="failed to get container status \"de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09\": rpc error: code = NotFound desc = could not find container \"de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09\": container with ID starting with de5f3dd41c13c9ef15b77a96472430a619c6afd57cc75e1d33c0af5d96ca0b09 not found: ID does not exist" Nov 01 09:11:20.259282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.259269 5710 scope.go:122] "RemoveContainer" containerID="2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210" Nov 01 09:11:20.269303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.269281 5710 scope.go:122] "RemoveContainer" containerID="2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210" Nov 01 09:11:20.269675 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:20.269647 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210\": container with ID starting with 2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210 not found: ID does not exist" containerID="2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210" Nov 01 09:11:20.269857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.269684 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210"} err="failed to get container status \"2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210\": rpc error: code = NotFound desc = could not find container \"2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210\": container with ID starting with 2af4de2f6d86140157eb017cf34df2f62c777f65f3d92aeedba964329d6e9210 not found: ID does not exist" Nov 01 09:11:20.381194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.381153 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68"] Nov 01 09:11:20.386032 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.385992 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaabf4-8fcc497b4-gvn68"] Nov 01 09:11:20.488410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.488320 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg"] Nov 01 09:11:20.492502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.492469 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaabf4-d8948c579-6k9zg"] Nov 01 09:11:20.555575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.555535 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:11:20.595102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.595056 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv"] Nov 01 09:11:20.597602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:20.597569 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaabf4-6d69848647-l4tqv"] Nov 01 09:11:21.058560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:21.058520 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346d880e-ef45-406b-9a77-80760c094c37" path="/var/lib/kubelet/pods/346d880e-ef45-406b-9a77-80760c094c37/volumes" Nov 01 09:11:21.059112 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:21.059093 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517fc65a-4127-4b76-946b-2b46862047d4" path="/var/lib/kubelet/pods/517fc65a-4127-4b76-946b-2b46862047d4/volumes" Nov 01 09:11:21.059474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:21.059461 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15350a6-9f01-49e9-a850-c210c8daaa2d" path="/var/lib/kubelet/pods/f15350a6-9f01-49e9-a850-c210c8daaa2d/volumes" Nov 01 09:11:28.998584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.998542 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd"] Nov 01 09:11:28.999481 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999459 5710 cpu_manager.go:414] "containerMap: removing container" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999482 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999503 5710 cpu_manager.go:414] "containerMap: removing container" podUID="346d880e-ef45-406b-9a77-80760c094c37" containerName="payload-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999510 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346d880e-ef45-406b-9a77-80760c094c37" containerName="payload-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999526 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b4628121-3162-4d72-a705-1f815ca00a56" containerName="payload-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999534 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4628121-3162-4d72-a705-1f815ca00a56" containerName="payload-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999558 5710 cpu_manager.go:414] "containerMap: removing container" podUID="10c9ac6c-1566-4f9d-8478-46fc13d615fa" containerName="payload-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999565 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c9ac6c-1566-4f9d-8478-46fc13d615fa" containerName="payload-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999582 5710 cpu_manager.go:414] "containerMap: removing container" podUID="03a163c1-ee6e-42b2-910e-81bf40f30097" containerName="payload-pre-processing" Nov 01 09:11:28.999595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999590 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a163c1-ee6e-42b2-910e-81bf40f30097" containerName="payload-pre-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999612 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9ea9f9c4-10f7-44c5-915e-c397521824ea" containerName="maas-api" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999620 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea9f9c4-10f7-44c5-915e-c397521824ea" containerName="maas-api" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999632 5710 cpu_manager.go:414] "containerMap: removing container" podUID="517fc65a-4127-4b76-946b-2b46862047d4" containerName="maas-api" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999639 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517fc65a-4127-4b76-946b-2b46862047d4" containerName="maas-api" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999661 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f15350a6-9f01-49e9-a850-c210c8daaa2d" containerName="payload-pre-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999672 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15350a6-9f01-49e9-a850-c210c8daaa2d" containerName="payload-pre-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999829 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15350a6-9f01-49e9-a850-c210c8daaa2d" containerName="payload-pre-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999851 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999860 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="346d880e-ef45-406b-9a77-80760c094c37" containerName="payload-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999871 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4628121-3162-4d72-a705-1f815ca00a56" containerName="payload-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999884 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c9ac6c-1566-4f9d-8478-46fc13d615fa" containerName="payload-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999894 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a163c1-ee6e-42b2-910e-81bf40f30097" containerName="payload-pre-processing" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999905 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea9f9c4-10f7-44c5-915e-c397521824ea" containerName="maas-api" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999915 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="517fc65a-4127-4b76-946b-2b46862047d4" containerName="maas-api" Nov 01 09:11:29.000106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:28.999925 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" Nov 01 09:11:29.005468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.005445 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.061777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.055349 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-55644e-openshift-default-dockercfg-gcpnz\"" Nov 01 09:11:29.092593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092553 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.092832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092636 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.092832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092700 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.092832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092758 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.093006 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092832 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.093006 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092912 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.093006 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.092986 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.093221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.093019 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.093221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.093087 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2fn\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.105206 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.105158 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd"] Nov 01 09:11:29.194186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194153 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194226 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194283 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194323 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194384 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194430 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194481 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194512 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194575 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2fn\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.194838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.194630 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.195296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.195268 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.195392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.195327 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.195392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.195354 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.195967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.195917 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.197822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.197615 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.198569 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.198540 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.252604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.252528 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.252784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.252663 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2fn\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn\") pod \"e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.320099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.320056 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:29.757185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.757145 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd"] Nov 01 09:11:29.765474 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:11:29.765437 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e16e5c_c01c_4246_a0f2_7f1179e5951c.slice/crio-e37ee007774ec4d2403bee4ac41ec157f4d5b8b0de9119aa5f24cc4d427fc647 WatchSource:0}: Error finding container e37ee007774ec4d2403bee4ac41ec157f4d5b8b0de9119aa5f24cc4d427fc647: Status 404 returned error can't find the container with id e37ee007774ec4d2403bee4ac41ec157f4d5b8b0de9119aa5f24cc4d427fc647 Nov 01 09:11:29.768165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.768130 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:11:29.768274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.768206 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:11:29.768274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:29.768236 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:11:30.297656 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:30.297621 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" event={"ID":"d2e16e5c-c01c-4246-a0f2-7f1179e5951c","Type":"ContainerStarted","Data":"b5ea5fbbdf83750a660d098b74d5fca04ce170de1858dc44302c34cdf043ea75"} Nov 01 09:11:30.298094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:30.297677 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" event={"ID":"d2e16e5c-c01c-4246-a0f2-7f1179e5951c","Type":"ContainerStarted","Data":"e37ee007774ec4d2403bee4ac41ec157f4d5b8b0de9119aa5f24cc4d427fc647"} Nov 01 09:11:30.321308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:30.321269 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:30.327764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:30.327703 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:30.414150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:30.414079 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podStartSLOduration=2.414062654 podStartE2EDuration="2.414062654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:11:30.411363769 +0000 UTC m=+2313.499700444" watchObservedRunningTime="2026-11-01 09:11:30.414062654 +0000 UTC m=+2313.502399326" Nov 01 09:11:31.304989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:31.304958 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:31.306003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:31.305979 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:11:36.687170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.687132 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb"] Nov 01 09:11:36.687884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.687864 5710 cpu_manager.go:414] "containerMap: removing container" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" Nov 01 09:11:36.687884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.687885 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd31e0d3-924d-400c-bb26-fccdcfd388b0" containerName="istio-proxy" Nov 01 09:11:36.691778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.691757 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.719227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.719184 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkj62\" (UniqueName: \"kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62\") pod \"payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.719386 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.719362 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.741465 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.741433 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-55644e\"" Nov 01 09:11:36.744822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.744770 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb"] Nov 01 09:11:36.787944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.787908 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-55644e-dockercfg-ps927\"" Nov 01 09:11:36.827066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.827025 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkj62\" (UniqueName: \"kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62\") pod \"payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.827304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.827222 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.828267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.828230 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.847296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.847255 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc"] Nov 01 09:11:36.852061 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.852037 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:36.886185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.886155 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkj62\" (UniqueName: \"kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62\") pod \"payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:36.904415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.904380 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc"] Nov 01 09:11:36.928859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.928822 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tmc\" (UniqueName: \"kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc\") pod \"payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:36.928859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:36.928863 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:37.008239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.008151 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:37.030311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.030273 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tmc\" (UniqueName: \"kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc\") pod \"payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:37.030466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.030332 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:37.031313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.031279 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:37.097173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.097142 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tmc\" (UniqueName: \"kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc\") pod \"payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:37.167080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.167035 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:37.389226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.389191 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb"] Nov 01 09:11:37.395247 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:11:37.395213 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4645f8_6919_4682_ad2d_738ad3dd6dae.slice/crio-cc0006132beb467380ac3fa943df05fc18661520e6142bfc70f939622cdb8e6a WatchSource:0}: Error finding container cc0006132beb467380ac3fa943df05fc18661520e6142bfc70f939622cdb8e6a: Status 404 returned error can't find the container with id cc0006132beb467380ac3fa943df05fc18661520e6142bfc70f939622cdb8e6a Nov 01 09:11:37.546506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:37.546469 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc"] Nov 01 09:11:37.553651 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:11:37.553623 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691b3adb_9c90_4da5_9327_0351757a7dcd.slice/crio-1746e876fa35b220918b9b336cd6c822df1b2fa97709c8e118fcaa7d0ce2d72b WatchSource:0}: Error finding container 1746e876fa35b220918b9b336cd6c822df1b2fa97709c8e118fcaa7d0ce2d72b: Status 404 returned error can't find the container with id 1746e876fa35b220918b9b336cd6c822df1b2fa97709c8e118fcaa7d0ce2d72b Nov 01 09:11:38.348352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.348310 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" event={"ID":"af4645f8-6919-4682-ad2d-738ad3dd6dae","Type":"ContainerStarted","Data":"06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd"} Nov 01 09:11:38.348839 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.348414 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" event={"ID":"af4645f8-6919-4682-ad2d-738ad3dd6dae","Type":"ContainerStarted","Data":"cc0006132beb467380ac3fa943df05fc18661520e6142bfc70f939622cdb8e6a"} Nov 01 09:11:38.348839 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.348524 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:38.350030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.350009 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" event={"ID":"691b3adb-9c90-4da5-9327-0351757a7dcd","Type":"ContainerStarted","Data":"389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d"} Nov 01 09:11:38.350083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.350049 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" event={"ID":"691b3adb-9c90-4da5-9327-0351757a7dcd","Type":"ContainerStarted","Data":"1746e876fa35b220918b9b336cd6c822df1b2fa97709c8e118fcaa7d0ce2d72b"} Nov 01 09:11:38.350118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.350096 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:38.459279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.459218 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" podStartSLOduration=2.459195109 podStartE2EDuration="2.459195109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:11:38.456527752 +0000 UTC m=+2321.544864440" watchObservedRunningTime="2026-11-01 09:11:38.459195109 +0000 UTC m=+2321.547531780" Nov 01 09:11:38.566298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:38.566238 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" podStartSLOduration=2.566224158 podStartE2EDuration="2.566224158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:11:38.56484231 +0000 UTC m=+2321.653178976" watchObservedRunningTime="2026-11-01 09:11:38.566224158 +0000 UTC m=+2321.654560829" Nov 01 09:11:43.327829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.327775 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq"] Nov 01 09:11:43.328368 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.328086 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" podUID="15118450-ccc6-470e-ba4b-bed1fdffa8c2" containerName="maas-api" containerID="cri-o://e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc" gracePeriod=30 Nov 01 09:11:43.639708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.639668 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:11:43.738434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.738390 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs\") pod \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " Nov 01 09:11:43.738667 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.738466 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\") pod \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " Nov 01 09:11:43.738667 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.738628 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca\") pod \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " Nov 01 09:11:43.738810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.738767 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp\" (UniqueName: \"kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp\") pod \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\" (UID: \"15118450-ccc6-470e-ba4b-bed1fdffa8c2\") " Nov 01 09:11:43.739355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.739308 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca" pod "15118450-ccc6-470e-ba4b-bed1fdffa8c2" (UID: "15118450-ccc6-470e-ba4b-bed1fdffa8c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:11:43.744829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.744785 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls" pod "15118450-ccc6-470e-ba4b-bed1fdffa8c2" (UID: "15118450-ccc6-470e-ba4b-bed1fdffa8c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:11:43.744829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.744811 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs" pod "15118450-ccc6-470e-ba4b-bed1fdffa8c2" (UID: "15118450-ccc6-470e-ba4b-bed1fdffa8c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:11:43.745273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.745243 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp" pod "15118450-ccc6-470e-ba4b-bed1fdffa8c2" (UID: "15118450-ccc6-470e-ba4b-bed1fdffa8c2"). InnerVolumeSpecName "kube-api-access-66kfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:11:43.840334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.840291 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:43.840334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.840329 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15118450-ccc6-470e-ba4b-bed1fdffa8c2-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:43.840334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.840339 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kfp\" (UniqueName: \"kubernetes.io/projected/15118450-ccc6-470e-ba4b-bed1fdffa8c2-kube-api-access-66kfp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:43.840597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:43.840349 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15118450-ccc6-470e-ba4b-bed1fdffa8c2-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:11:44.391407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.391368 5710 generic.go:356] "Generic (PLEG): container finished" podID="15118450-ccc6-470e-ba4b-bed1fdffa8c2" containerID="e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc" exitCode=0 Nov 01 09:11:44.391949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.391414 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" Nov 01 09:11:44.391949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.391454 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" event={"ID":"15118450-ccc6-470e-ba4b-bed1fdffa8c2","Type":"ContainerDied","Data":"e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc"} Nov 01 09:11:44.391949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.391521 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq" event={"ID":"15118450-ccc6-470e-ba4b-bed1fdffa8c2","Type":"ContainerDied","Data":"d5d7fa9c87342467a5b6ea960d328ef6acffef5f1d18c086a9f7cd13af725f30"} Nov 01 09:11:44.391949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.391554 5710 scope.go:122] "RemoveContainer" containerID="e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc" Nov 01 09:11:44.402741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.402673 5710 scope.go:122] "RemoveContainer" containerID="e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc" Nov 01 09:11:44.403054 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:44.403028 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc\": container with ID starting with e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc not found: ID does not exist" containerID="e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc" Nov 01 09:11:44.403178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.403066 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc"} err="failed to get container status \"e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc\": rpc error: code = NotFound desc = could not find container \"e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc\": container with ID starting with e8654558c973ea8d938762167eeed3be38972e3a44a26584076ba5a8357491bc not found: ID does not exist" Nov 01 09:11:44.546177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.546134 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq"] Nov 01 09:11:44.592793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.592754 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5fe64d-68cd58484c-knrrq"] Nov 01 09:11:44.935573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.935532 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv"] Nov 01 09:11:44.936238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.936223 5710 cpu_manager.go:414] "containerMap: removing container" podUID="15118450-ccc6-470e-ba4b-bed1fdffa8c2" containerName="maas-api" Nov 01 09:11:44.936284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.936241 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15118450-ccc6-470e-ba4b-bed1fdffa8c2" containerName="maas-api" Nov 01 09:11:44.936395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.936365 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="15118450-ccc6-470e-ba4b-bed1fdffa8c2" containerName="maas-api" Nov 01 09:11:44.941386 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.941361 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:44.991013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.990975 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9f4e9b-serving-cert\"" Nov 01 09:11:44.994508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:44.994476 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv"] Nov 01 09:11:45.059880 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.059844 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15118450-ccc6-470e-ba4b-bed1fdffa8c2" path="/var/lib/kubelet/pods/15118450-ccc6-470e-ba4b-bed1fdffa8c2/volumes" Nov 01 09:11:45.067236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.067183 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.067389 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.067263 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6bl\" (UniqueName: \"kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.067389 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.067351 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.067490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.067419 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.173842 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.173924 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6bl\" (UniqueName: \"kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.173964 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.174100 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:45.177410 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-serving-cert: secret "maas-api-e2e-praxis-b-9f4e9b-serving-cert" not found Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:45.177505 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls podName:b04181e8-dffe-4c5d-b894-2e1a74914510 nodeName:}" failed. No retries permitted until 2026-11-01 09:11:45.677478164 +0000 UTC m=+2328.765814830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls") pod "maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" (UID: "b04181e8-dffe-4c5d-b894-2e1a74914510") : secret "maas-api-e2e-praxis-b-9f4e9b-serving-cert" not found Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.183149 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.183518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.183518 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.242042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.241959 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6bl\" (UniqueName: \"kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.478640 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:11:45.478595 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:11:45.681522 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.681475 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.684261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.684228 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:45.856071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:45.856041 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:46.239021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:46.238971 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv"] Nov 01 09:11:46.246704 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:11:46.246670 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04181e8_dffe_4c5d_b894_2e1a74914510.slice/crio-884f84fd9c82ac4fb25666fbb69c8bae6cd91fb6f084d8abb31fa1f1e660a425 WatchSource:0}: Error finding container 884f84fd9c82ac4fb25666fbb69c8bae6cd91fb6f084d8abb31fa1f1e660a425: Status 404 returned error can't find the container with id 884f84fd9c82ac4fb25666fbb69c8bae6cd91fb6f084d8abb31fa1f1e660a425 Nov 01 09:11:46.248533 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:46.248506 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:11:46.407969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:46.407931 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" event={"ID":"b04181e8-dffe-4c5d-b894-2e1a74914510","Type":"ContainerStarted","Data":"884f84fd9c82ac4fb25666fbb69c8bae6cd91fb6f084d8abb31fa1f1e660a425"} Nov 01 09:11:47.417401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:47.417342 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" event={"ID":"b04181e8-dffe-4c5d-b894-2e1a74914510","Type":"ContainerStarted","Data":"ffdfb0f942987affc0d9bc84d5bf30546ad72b6dea7ad499599f86b58f14c7e2"} Nov 01 09:11:47.417924 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:47.417585 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:47.550525 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:47.550416 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" podStartSLOduration=2.542353271 podStartE2EDuration="3.550398461s" totalImagesPullingTime="1.00804519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:11:47.547396941 +0000 UTC m=+2330.635733614" watchObservedRunningTime="2026-11-01 09:11:47.550398461 +0000 UTC m=+2330.638735132" Nov 01 09:11:49.359248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:49.359218 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:11:49.359681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:49.359595 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:11:49.983131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:49.983103 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx_d72205af-d592-4096-aba0-24efe89c01ee/payload-processing/0.log" Nov 01 09:11:50.275500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:50.275420 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sccmk_1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378/payload-processing/0.log" Nov 01 09:11:50.513645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:50.513615 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-sccmk_1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378/payload-processing/0.log" Nov 01 09:11:50.721346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:50.721318 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf_9da49772-4bf4-42eb-9a19-8b91024cc1e2/payload-processing/0.log" Nov 01 09:11:50.989509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:50.989429 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf_9da49772-4bf4-42eb-9a19-8b91024cc1e2/payload-processing/0.log" Nov 01 09:11:53.430317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:53.430284 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:11:54.458801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.458766 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr"] Nov 01 09:11:54.464002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.463978 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.528152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.528116 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-28efa1-openshift-default-dockercfg-xvx7p\"" Nov 01 09:11:54.588333 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.588291 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr"] Nov 01 09:11:54.612540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.612501 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.612873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.612835 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq2t\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.612993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.612983 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.613053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.613006 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.613053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.613041 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.613171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.613062 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.613171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.613101 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.613171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.613131 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.613171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.613158 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714272 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714327 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714367 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714396 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714448 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714488 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714526 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.714633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714607 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.715001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714638 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq2t\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.715001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.714891 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.715143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.715111 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.715218 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.715175 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.715267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.715217 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.715553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.715532 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.717682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.717659 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.717936 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.717913 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.772735 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.772681 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.772735 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.772712 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq2t\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t\") pod \"e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:54.777967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:54.777932 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:55.231377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.231342 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr"] Nov 01 09:11:55.238893 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:11:55.238860 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe63c5e_6364_43ed_9d85_781e4f30ab3a.slice/crio-804071d02176e1b07c27e4fc7744bf9ad0c70441556475a7f7fca0e9c2b97d83 WatchSource:0}: Error finding container 804071d02176e1b07c27e4fc7744bf9ad0c70441556475a7f7fca0e9c2b97d83: Status 404 returned error can't find the container with id 804071d02176e1b07c27e4fc7744bf9ad0c70441556475a7f7fca0e9c2b97d83 Nov 01 09:11:55.241578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.241546 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:11:55.241652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.241613 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:11:55.241652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.241644 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:11:55.474982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.474939 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" event={"ID":"7fe63c5e-6364-43ed-9d85-781e4f30ab3a","Type":"ContainerStarted","Data":"9b4fcd38b2a866e5731c5a95cb70b4246c5b90c2e503df54e8a95e090597a183"} Nov 01 09:11:55.475616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.475029 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" event={"ID":"7fe63c5e-6364-43ed-9d85-781e4f30ab3a","Type":"ContainerStarted","Data":"804071d02176e1b07c27e4fc7744bf9ad0c70441556475a7f7fca0e9c2b97d83"} Nov 01 09:11:55.598070 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.597998 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podStartSLOduration=1.5979747610000001 podStartE2EDuration="1.597974761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:11:55.593448467 +0000 UTC m=+2338.681785140" watchObservedRunningTime="2026-11-01 09:11:55.597974761 +0000 UTC m=+2338.686311433" Nov 01 09:11:55.778092 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:55.778049 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:56.785014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:56.784983 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:56.799635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:56.799602 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq"] Nov 01 09:11:57.489118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:57.489082 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:11:57.490169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:11:57.490146 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:12:02.049741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.049566 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2"] Nov 01 09:12:02.068990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.068958 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.081668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.081628 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2"] Nov 01 09:12:02.117903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.117859 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-28efa1\"" Nov 01 09:12:02.118142 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.117995 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-28efa1-dockercfg-kc6x7\"" Nov 01 09:12:02.130697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.130662 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.130875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.130839 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkw5\" (UniqueName: \"kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5\") pod \"payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.231661 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.231625 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.231892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.231773 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkw5\" (UniqueName: \"kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5\") pod \"payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.232380 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.232359 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.289679 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.289643 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkw5\" (UniqueName: \"kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5\") pod \"payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.387790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.387750 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:02.770153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:02.770075 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2"] Nov 01 09:12:02.777376 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:12:02.777340 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036a5542_40c4_4f8a_9dbf_bd611c41f0f5.slice/crio-3689a2f33e74be6d0dd965b00d9f17f38102d2d15228fd0b10c697e4ad3e7366 WatchSource:0}: Error finding container 3689a2f33e74be6d0dd965b00d9f17f38102d2d15228fd0b10c697e4ad3e7366: Status 404 returned error can't find the container with id 3689a2f33e74be6d0dd965b00d9f17f38102d2d15228fd0b10c697e4ad3e7366 Nov 01 09:12:03.534347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:03.534313 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" event={"ID":"036a5542-40c4-4f8a-9dbf-bd611c41f0f5","Type":"ContainerStarted","Data":"d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53"} Nov 01 09:12:03.534848 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:03.534365 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" event={"ID":"036a5542-40c4-4f8a-9dbf-bd611c41f0f5","Type":"ContainerStarted","Data":"3689a2f33e74be6d0dd965b00d9f17f38102d2d15228fd0b10c697e4ad3e7366"} Nov 01 09:12:03.534848 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:03.534505 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:03.643893 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:03.643829 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" podStartSLOduration=2.643813517 podStartE2EDuration="2.643813517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:03.642451586 +0000 UTC m=+2346.730788258" watchObservedRunningTime="2026-11-01 09:12:03.643813517 +0000 UTC m=+2346.732150188" Nov 01 09:12:11.411642 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:11.411598 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:12:11.822535 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:11.822426 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" podUID="f352dd2f-936b-400e-820b-7dcf9b4aaa4b" containerName="main" containerID="cri-o://ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e" gracePeriod=45 Nov 01 09:12:12.175852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.174010 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:12:12.313644 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313606 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm\") pod \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " Nov 01 09:12:12.313863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313704 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache\") pod \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " Nov 01 09:12:12.313863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313798 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir\") pod \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " Nov 01 09:12:12.313863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313839 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs\") pod \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " Nov 01 09:12:12.313998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313923 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq\" (UniqueName: \"kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq\") pod \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " Nov 01 09:12:12.313998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313961 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home\") pod \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\" (UID: \"f352dd2f-936b-400e-820b-7dcf9b4aaa4b\") " Nov 01 09:12:12.313998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.313970 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache" pod "f352dd2f-936b-400e-820b-7dcf9b4aaa4b" (UID: "f352dd2f-936b-400e-820b-7dcf9b4aaa4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:12:12.314231 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.314208 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home" pod "f352dd2f-936b-400e-820b-7dcf9b4aaa4b" (UID: "f352dd2f-936b-400e-820b-7dcf9b4aaa4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:12:12.314397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.314374 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir" pod "f352dd2f-936b-400e-820b-7dcf9b4aaa4b" (UID: "f352dd2f-936b-400e-820b-7dcf9b4aaa4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:12:12.314628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.314610 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:12:12.314705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.314632 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:12:12.314705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.314647 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:12:12.316888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.316855 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq" pod "f352dd2f-936b-400e-820b-7dcf9b4aaa4b" (UID: "f352dd2f-936b-400e-820b-7dcf9b4aaa4b"). InnerVolumeSpecName "kube-api-access-s6dnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:12:12.317818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.317791 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs" pod "f352dd2f-936b-400e-820b-7dcf9b4aaa4b" (UID: "f352dd2f-936b-400e-820b-7dcf9b4aaa4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:12:12.317915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.317900 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm" pod "f352dd2f-936b-400e-820b-7dcf9b4aaa4b" (UID: "f352dd2f-936b-400e-820b-7dcf9b4aaa4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:12:12.418957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.416750 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:12:12.418957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.416795 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dnq\" (UniqueName: \"kubernetes.io/projected/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-kube-api-access-s6dnq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:12:12.418957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.416811 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f352dd2f-936b-400e-820b-7dcf9b4aaa4b-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:12:12.600016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.599975 5710 generic.go:356] "Generic (PLEG): container finished" podID="f352dd2f-936b-400e-820b-7dcf9b4aaa4b" containerID="ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e" exitCode=0 Nov 01 09:12:12.600217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.600027 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" Nov 01 09:12:12.600217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.600052 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" event={"ID":"f352dd2f-936b-400e-820b-7dcf9b4aaa4b","Type":"ContainerDied","Data":"ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e"} Nov 01 09:12:12.600217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.600104 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq" event={"ID":"f352dd2f-936b-400e-820b-7dcf9b4aaa4b","Type":"ContainerDied","Data":"d01b5e0dcd7d1611e3d6d5fee05c68d0b51d46a744e5dde9e4a5f83ae40359da"} Nov 01 09:12:12.600217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.600121 5710 scope.go:122] "RemoveContainer" containerID="ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e" Nov 01 09:12:12.613003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.612979 5710 scope.go:122] "RemoveContainer" containerID="ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e" Nov 01 09:12:12.613362 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:12:12.613339 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e\": container with ID starting with ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e not found: ID does not exist" containerID="ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e" Nov 01 09:12:12.613417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.613374 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e"} err="failed to get container status \"ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e\": rpc error: code = NotFound desc = could not find container \"ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e\": container with ID starting with ea917ca7517e56287fee4780ed559843d14e6855b6c24fc8cb0bdf2607516f0e not found: ID does not exist" Nov 01 09:12:12.755023 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.754988 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq"] Nov 01 09:12:12.801303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:12.801262 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b893c8/ipp-route-b893c8-kserve-5fcf9b44d9-hjbtq"] Nov 01 09:12:13.058914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:13.058836 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f352dd2f-936b-400e-820b-7dcf9b4aaa4b" path="/var/lib/kubelet/pods/f352dd2f-936b-400e-820b-7dcf9b4aaa4b/volumes" Nov 01 09:12:14.556761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.552354 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:12:14.865940 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.865874 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr"] Nov 01 09:12:14.866959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.866924 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f352dd2f-936b-400e-820b-7dcf9b4aaa4b" containerName="main" Nov 01 09:12:14.873823 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.873775 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f352dd2f-936b-400e-820b-7dcf9b4aaa4b" containerName="main" Nov 01 09:12:14.878909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.878875 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f352dd2f-936b-400e-820b-7dcf9b4aaa4b" containerName="main" Nov 01 09:12:14.892041 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.892003 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr"] Nov 01 09:12:14.892547 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.892427 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:14.963039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.963005 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6752e4\"" Nov 01 09:12:14.963250 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.963103 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6752e4-dockercfg-xmlrs\"" Nov 01 09:12:14.968543 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.968512 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:14.968787 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:14.968765 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdccz\" (UniqueName: \"kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz\") pod \"payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:15.069813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:15.069770 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdccz\" (UniqueName: \"kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz\") pod \"payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:15.070018 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:15.069882 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:15.070601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:15.070580 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:15.209869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:15.209778 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdccz\" (UniqueName: \"kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz\") pod \"payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:15.236829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:15.236793 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:15.506826 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:12:15.506487 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:12:15.641200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:15.641162 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr"] Nov 01 09:12:15.648080 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:12:15.648046 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode279588d_21f3_41b5_88a5_676361e5e2a8.slice/crio-e3a2ea0d0866ff712a34855bb22d8e91da67eb12d08e5e7b2aa388c7ecbf63d9 WatchSource:0}: Error finding container e3a2ea0d0866ff712a34855bb22d8e91da67eb12d08e5e7b2aa388c7ecbf63d9: Status 404 returned error can't find the container with id e3a2ea0d0866ff712a34855bb22d8e91da67eb12d08e5e7b2aa388c7ecbf63d9 Nov 01 09:12:16.637927 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:16.637888 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" event={"ID":"e279588d-21f3-41b5-88a5-676361e5e2a8","Type":"ContainerStarted","Data":"fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1"} Nov 01 09:12:16.638128 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:16.637955 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" event={"ID":"e279588d-21f3-41b5-88a5-676361e5e2a8","Type":"ContainerStarted","Data":"e3a2ea0d0866ff712a34855bb22d8e91da67eb12d08e5e7b2aa388c7ecbf63d9"} Nov 01 09:12:16.638128 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:16.637986 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:16.765738 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:16.765672 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" podStartSLOduration=2.76565715 podStartE2EDuration="2.76565715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:12:16.763999296 +0000 UTC m=+2359.852335970" watchObservedRunningTime="2026-11-01 09:12:16.76565715 +0000 UTC m=+2359.853993822" Nov 01 09:12:21.915024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:21.914964 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:12:21.915468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:21.915059 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:12:22.682394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:22.682356 5710 generic.go:356] "Generic (PLEG): container finished" podID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerID="e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603" exitCode=0 Nov 01 09:12:22.682642 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:22.682428 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" event={"ID":"e1dbd1a1-748a-4956-86db-8d248a00053f","Type":"ContainerDied","Data":"e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603"} Nov 01 09:12:22.682897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:22.682883 5710 scope.go:122] "RemoveContainer" containerID="e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603" Nov 01 09:12:22.684599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:22.684568 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:12:22.684771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:22.684646 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:12:22.684771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:22.684683 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:12:23.711744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:23.711386 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" event={"ID":"e1dbd1a1-748a-4956-86db-8d248a00053f","Type":"ContainerStarted","Data":"618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823"} Nov 01 09:12:24.674010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:24.673969 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:12:24.674265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:24.674236 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 09:12:24.674367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:24.674317 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 09:12:25.673860 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:25.673815 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 09:12:25.674309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:25.673877 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 09:12:26.674336 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:26.674292 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 09:12:26.674800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:26.674354 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 09:12:27.651344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:27.651305 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:12:27.674226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:27.674183 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 09:12:27.674379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:27.674246 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 09:12:28.673557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:28.673516 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 09:12:28.673816 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:28.673577 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 09:12:29.678840 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:29.678801 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:12:29.679346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:29.679169 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:12:29.680173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:29.680146 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:12:45.479851 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:12:45.479805 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:12:54.148011 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:12:54.147966 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:13:00.960765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:00.960655 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:13:00.974592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:00.974561 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:13:02.348767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:02.347921 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:13:02.348767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:02.347999 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:13:05.005354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:05.005316 5710 generic.go:356] "Generic (PLEG): container finished" podID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerID="b5ea5fbbdf83750a660d098b74d5fca04ce170de1858dc44302c34cdf043ea75" exitCode=0 Nov 01 09:13:05.005874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:05.005386 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" event={"ID":"d2e16e5c-c01c-4246-a0f2-7f1179e5951c","Type":"ContainerDied","Data":"b5ea5fbbdf83750a660d098b74d5fca04ce170de1858dc44302c34cdf043ea75"} Nov 01 09:13:05.005926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:05.005874 5710 scope.go:122] "RemoveContainer" containerID="b5ea5fbbdf83750a660d098b74d5fca04ce170de1858dc44302c34cdf043ea75" Nov 01 09:13:05.007590 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:05.007560 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:13:05.007672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:05.007645 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:13:05.007751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:05.007679 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:13:06.015973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:06.015927 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" event={"ID":"d2e16e5c-c01c-4246-a0f2-7f1179e5951c","Type":"ContainerStarted","Data":"4ad6a820ceb82c1f4248bee29973cc44475dfb51b55d8801876c78b8ecd7f175"} Nov 01 09:13:06.320525 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:06.320428 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:13:06.320761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:06.320698 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 09:13:06.320827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:06.320806 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 09:13:07.322870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:07.322826 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 09:13:07.323341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:07.322898 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 09:13:08.320693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:08.320649 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 09:13:08.320906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:08.320711 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 09:13:09.321210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:09.321167 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 09:13:09.321664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:09.321232 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 09:13:10.326752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:10.325304 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 09:13:10.326752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:10.325383 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 09:13:11.321401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:11.321356 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 09:13:11.321611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:11.321433 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 09:13:12.325164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:12.325133 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:13:12.325615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:12.325542 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:13:12.326382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:12.326364 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:13:15.480542 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:15.480501 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:13:22.507984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.507944 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5"] Nov 01 09:13:22.512678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.512652 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.568364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.568326 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-6752e4-serving-cert\"" Nov 01 09:13:22.571792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.571710 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5"] Nov 01 09:13:22.598317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.598274 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.598508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.598454 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.598508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.598494 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxln\" (UniqueName: \"kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.598597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.598529 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.636906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.636867 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:13:22.700403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.700363 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.700605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.700520 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.700605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.700558 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxln\" (UniqueName: \"kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.700605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.700590 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.700951 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:22.700650 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-serving-cert: secret "maas-api-e2e-praxis-a-6752e4-serving-cert" not found Nov 01 09:13:22.700951 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:22.700779 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls podName:57ab4c3c-3c9b-4844-a6ab-02fb508c750b nodeName:}" failed. No retries permitted until 2026-11-01 09:13:23.200753327 +0000 UTC m=+2426.289089995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls") pod "maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" (UID: "57ab4c3c-3c9b-4844-a6ab-02fb508c750b") : secret "maas-api-e2e-praxis-a-6752e4-serving-cert" not found Nov 01 09:13:22.704253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.704221 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.709354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.709325 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:22.760557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:22.760481 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxln\" (UniqueName: \"kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:23.216170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:23.216114 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:23.218957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:23.218928 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:23.431316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:23.431272 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:23.875311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:23.875267 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5"] Nov 01 09:13:23.879770 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:13:23.879732 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ab4c3c_3c9b_4844_a6ab_02fb508c750b.slice/crio-8963e14249ffb6ac8c205d38b76096b608b970597ba1ccfe1b1f466550feef8d WatchSource:0}: Error finding container 8963e14249ffb6ac8c205d38b76096b608b970597ba1ccfe1b1f466550feef8d: Status 404 returned error can't find the container with id 8963e14249ffb6ac8c205d38b76096b608b970597ba1ccfe1b1f466550feef8d Nov 01 09:13:24.139619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:24.139527 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" event={"ID":"57ab4c3c-3c9b-4844-a6ab-02fb508c750b","Type":"ContainerStarted","Data":"8963e14249ffb6ac8c205d38b76096b608b970597ba1ccfe1b1f466550feef8d"} Nov 01 09:13:25.157088 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:25.156986 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" event={"ID":"57ab4c3c-3c9b-4844-a6ab-02fb508c750b","Type":"ContainerStarted","Data":"d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40"} Nov 01 09:13:25.157670 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:25.157647 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:25.271266 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:25.271190 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" podStartSLOduration=2.2968937609999998 podStartE2EDuration="3.271168804s" totalImagesPullingTime="974.275043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:13:25.26739063 +0000 UTC m=+2428.355727304" watchObservedRunningTime="2026-11-01 09:13:25.271168804 +0000 UTC m=+2428.359505482" Nov 01 09:13:28.170685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:28.170647 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq"] Nov 01 09:13:28.171225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:28.170897 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" containerID="cri-o://20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32" gracePeriod=30 Nov 01 09:13:31.170205 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:31.170174 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:13:31.741088 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:31.741047 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:13:31.741254 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:31.741114 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:13:32.373898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:32.373863 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5"] Nov 01 09:13:32.374396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:32.374084 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" containerID="cri-o://566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846" gracePeriod=30 Nov 01 09:13:33.448307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.448240 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 09:13:33.597308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597267 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597364 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597385 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597433 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597490 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597513 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597581 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597604 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597596 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:33.597701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597641 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert\") pod \"0e4eb97f-3177-47da-befe-2cde7b743657\" (UID: \"0e4eb97f-3177-47da-befe-2cde7b743657\") " Nov 01 09:13:33.597701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.597690 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:33.598210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598124 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:33.598210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598179 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:13:33.598210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598184 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:33.598574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598553 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.598649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598581 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4eb97f-3177-47da-befe-2cde7b743657-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.598649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598598 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.598649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598612 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.598649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.598628 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.600262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.600221 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:33.600262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.600230 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:13:33.600430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.600323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:13:33.600430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.600352 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg" pod "0e4eb97f-3177-47da-befe-2cde7b743657" (UID: "0e4eb97f-3177-47da-befe-2cde7b743657"). InnerVolumeSpecName "kube-api-access-h96kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:13:33.699810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.699664 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96kg\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-kube-api-access-h96kg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.699810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.699704 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4eb97f-3177-47da-befe-2cde7b743657-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.699810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.699735 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4eb97f-3177-47da-befe-2cde7b743657-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:33.699810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:33.699749 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4eb97f-3177-47da-befe-2cde7b743657-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:34.234838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.234754 5710 generic.go:356] "Generic (PLEG): container finished" podID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerID="9b4fcd38b2a866e5731c5a95cb70b4246c5b90c2e503df54e8a95e090597a183" exitCode=0 Nov 01 09:13:34.235030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.234827 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" event={"ID":"7fe63c5e-6364-43ed-9d85-781e4f30ab3a","Type":"ContainerDied","Data":"9b4fcd38b2a866e5731c5a95cb70b4246c5b90c2e503df54e8a95e090597a183"} Nov 01 09:13:34.235378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.235356 5710 scope.go:122] "RemoveContainer" containerID="9b4fcd38b2a866e5731c5a95cb70b4246c5b90c2e503df54e8a95e090597a183" Nov 01 09:13:34.237219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237192 5710 generic.go:356] "Generic (PLEG): container finished" podID="0e4eb97f-3177-47da-befe-2cde7b743657" containerID="20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32" exitCode=0 Nov 01 09:13:34.237219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237214 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" event={"ID":"0e4eb97f-3177-47da-befe-2cde7b743657","Type":"ContainerDied","Data":"20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32"} Nov 01 09:13:34.237403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237255 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" Nov 01 09:13:34.237403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237271 5710 scope.go:122] "RemoveContainer" containerID="20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32" Nov 01 09:13:34.237403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237190 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:13:34.237403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237357 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:13:34.237403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237259 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq" event={"ID":"0e4eb97f-3177-47da-befe-2cde7b743657","Type":"ContainerDied","Data":"85f79c9413a25d82fa243690714d43ba1e045a1bc13c746829f67cadab8bbba0"} Nov 01 09:13:34.237645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.237410 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:13:34.253150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.253121 5710 scope.go:122] "RemoveContainer" containerID="aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0" Nov 01 09:13:34.263970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.263941 5710 scope.go:122] "RemoveContainer" containerID="20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32" Nov 01 09:13:34.264400 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:34.264367 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32\": container with ID starting with 20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32 not found: ID does not exist" containerID="20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32" Nov 01 09:13:34.264501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.264417 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32"} err="failed to get container status \"20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32\": rpc error: code = NotFound desc = could not find container \"20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32\": container with ID starting with 20531951b1afa955a7886f4d52bdfd56d01f8baf8050563574ffa4404103df32 not found: ID does not exist" Nov 01 09:13:34.264501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.264442 5710 scope.go:122] "RemoveContainer" containerID="aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0" Nov 01 09:13:34.264838 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:34.264816 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0\": container with ID starting with aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0 not found: ID does not exist" containerID="aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0" Nov 01 09:13:34.264945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.264844 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0"} err="failed to get container status \"aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0\": rpc error: code = NotFound desc = could not find container \"aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0\": container with ID starting with aa5b8e961e922e79ab871185490fe526bb9b2ae12d34e650fcd0092f4f0377e0 not found: ID does not exist" Nov 01 09:13:34.503863 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.503649 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq"] Nov 01 09:13:34.511288 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:34.511250 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2xbbq"] Nov 01 09:13:35.058552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:35.058514 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" path="/var/lib/kubelet/pods/0e4eb97f-3177-47da-befe-2cde7b743657/volumes" Nov 01 09:13:35.245771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:35.245704 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" event={"ID":"7fe63c5e-6364-43ed-9d85-781e4f30ab3a","Type":"ContainerStarted","Data":"80df6e6c7c6a1637ed19546ff4e5aabf2d9f7154b83ff48dfd6ce302141c2aa9"} Nov 01 09:13:35.778999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:35.778961 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:13:35.779541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:35.779400 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 09:13:35.779541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:35.779475 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 09:13:36.778586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:36.778540 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 09:13:36.778870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:36.778614 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 09:13:37.722457 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.722424 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 09:13:37.748504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748461 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748514 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748586 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748618 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748680 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748767 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748802 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.748991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.748859 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.749140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.749004 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket\") pod \"b31ff5cf-7c02-470b-b693-6a426ef895ec\" (UID: \"b31ff5cf-7c02-470b-b693-6a426ef895ec\") " Nov 01 09:13:37.750028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.749705 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:37.753111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.750113 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:13:37.753111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.750415 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:37.753111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.751977 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:37.753627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.753478 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:37.755911 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.755870 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:13:37.756807 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.756778 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:13:37.757465 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.757171 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "kube-api-access-g82d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:13:37.757585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.757566 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo" pod "b31ff5cf-7c02-470b-b693-6a426ef895ec" (UID: "b31ff5cf-7c02-470b-b693-6a426ef895ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:13:37.778523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.778484 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 09:13:37.778696 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.778545 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 09:13:37.851058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851019 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851059 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851075 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851084 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851119 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851130 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851138 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31ff5cf-7c02-470b-b693-6a426ef895ec-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851146 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31ff5cf-7c02-470b-b693-6a426ef895ec-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:37.851294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:37.851155 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82d5\" (UniqueName: \"kubernetes.io/projected/b31ff5cf-7c02-470b-b693-6a426ef895ec-kube-api-access-g82d5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:38.266834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.266795 5710 generic.go:356] "Generic (PLEG): container finished" podID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerID="566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846" exitCode=0 Nov 01 09:13:38.267038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.266846 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" Nov 01 09:13:38.267038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.266864 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" event={"ID":"b31ff5cf-7c02-470b-b693-6a426ef895ec","Type":"ContainerDied","Data":"566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846"} Nov 01 09:13:38.267038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.266927 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5" event={"ID":"b31ff5cf-7c02-470b-b693-6a426ef895ec","Type":"ContainerDied","Data":"e4f27217d6b47bf4982c4963f3f243e87529ded15c77ce2fdac9b8d32de3c091"} Nov 01 09:13:38.267038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.266957 5710 scope.go:122] "RemoveContainer" containerID="566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846" Nov 01 09:13:38.278985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.278962 5710 scope.go:122] "RemoveContainer" containerID="566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846" Nov 01 09:13:38.279350 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:38.279329 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846\": container with ID starting with 566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846 not found: ID does not exist" containerID="566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846" Nov 01 09:13:38.279423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.279363 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846"} err="failed to get container status \"566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846\": rpc error: code = NotFound desc = could not find container \"566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846\": container with ID starting with 566e1ba34112bc17dc2858c2b2876eede11e806e84c429da8867cb769b825846 not found: ID does not exist" Nov 01 09:13:38.421617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.421582 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5"] Nov 01 09:13:38.467974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.467937 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-xk6c5"] Nov 01 09:13:38.779394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.779348 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 09:13:38.779889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:38.779404 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 09:13:39.059332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:39.059244 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" path="/var/lib/kubelet/pods/b31ff5cf-7c02-470b-b693-6a426ef895ec/volumes" Nov 01 09:13:39.779075 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:39.779033 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 09:13:39.779292 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:39.779092 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 09:13:40.783853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:40.783820 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:13:40.784257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:40.784140 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:13:40.784895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:40.784877 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:13:45.479699 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:45.479660 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:13:48.225329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.225288 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w"] Nov 01 09:13:48.225980 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.225563 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" podUID="93935dae-3a7a-4ff3-a4fb-f193691b4147" containerName="maas-api" containerID="cri-o://350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83" gracePeriod=30 Nov 01 09:13:48.563939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.563913 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:13:48.706369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.706331 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca\") pod \"93935dae-3a7a-4ff3-a4fb-f193691b4147\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " Nov 01 09:13:48.706559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.706421 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs\") pod \"93935dae-3a7a-4ff3-a4fb-f193691b4147\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " Nov 01 09:13:48.706559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.706520 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm\" (UniqueName: \"kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm\") pod \"93935dae-3a7a-4ff3-a4fb-f193691b4147\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " Nov 01 09:13:48.706796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.706764 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca" pod "93935dae-3a7a-4ff3-a4fb-f193691b4147" (UID: "93935dae-3a7a-4ff3-a4fb-f193691b4147"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:13:48.706905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.706775 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\") pod \"93935dae-3a7a-4ff3-a4fb-f193691b4147\" (UID: \"93935dae-3a7a-4ff3-a4fb-f193691b4147\") " Nov 01 09:13:48.707650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.707620 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93935dae-3a7a-4ff3-a4fb-f193691b4147-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:48.708994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.708963 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm" pod "93935dae-3a7a-4ff3-a4fb-f193691b4147" (UID: "93935dae-3a7a-4ff3-a4fb-f193691b4147"). InnerVolumeSpecName "kube-api-access-8wksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:13:48.709123 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.709091 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls" pod "93935dae-3a7a-4ff3-a4fb-f193691b4147" (UID: "93935dae-3a7a-4ff3-a4fb-f193691b4147"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:13:48.709413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.709383 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs" pod "93935dae-3a7a-4ff3-a4fb-f193691b4147" (UID: "93935dae-3a7a-4ff3-a4fb-f193691b4147"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:13:48.809047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.808949 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wksm\" (UniqueName: \"kubernetes.io/projected/93935dae-3a7a-4ff3-a4fb-f193691b4147-kube-api-access-8wksm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:48.809307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.809284 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:48.809445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:48.809421 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93935dae-3a7a-4ff3-a4fb-f193691b4147-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:13:49.343939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.343907 5710 generic.go:356] "Generic (PLEG): container finished" podID="93935dae-3a7a-4ff3-a4fb-f193691b4147" containerID="350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83" exitCode=0 Nov 01 09:13:49.344464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.343954 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" Nov 01 09:13:49.344464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.343993 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" event={"ID":"93935dae-3a7a-4ff3-a4fb-f193691b4147","Type":"ContainerDied","Data":"350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83"} Nov 01 09:13:49.344464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.344058 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w" event={"ID":"93935dae-3a7a-4ff3-a4fb-f193691b4147","Type":"ContainerDied","Data":"7abd7e3b70a47c09e7fc019fb2d3cd3d78cd1d613616d1bb2414959a3249c8ae"} Nov 01 09:13:49.344464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.344091 5710 scope.go:122] "RemoveContainer" containerID="350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83" Nov 01 09:13:49.366444 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.366415 5710 scope.go:122] "RemoveContainer" containerID="350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83" Nov 01 09:13:49.366813 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:13:49.366788 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83\": container with ID starting with 350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83 not found: ID does not exist" containerID="350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83" Nov 01 09:13:49.366871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.366825 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83"} err="failed to get container status \"350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83\": rpc error: code = NotFound desc = could not find container \"350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83\": container with ID starting with 350662e34445854ea78195e16083f46c8dfab930348dbf9846d488ec5c6c6e83 not found: ID does not exist" Nov 01 09:13:49.498469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.498420 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w"] Nov 01 09:13:49.545126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:49.545086 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-616efa-579fb4b7f-cd47w"] Nov 01 09:13:50.146399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:50.146362 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:13:51.059541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:13:51.059502 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93935dae-3a7a-4ff3-a4fb-f193691b4147" path="/var/lib/kubelet/pods/93935dae-3a7a-4ff3-a4fb-f193691b4147/volumes" Nov 01 09:14:00.264660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:00.264570 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5"] Nov 01 09:14:00.361458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:00.361417 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj"] Nov 01 09:14:05.265311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.265263 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" podUID="26846be6-9903-4d21-9e70-e32de7fbe366" containerName="payload-processing" containerID="cri-o://596796f7335ca4483347338df87bd4f8a9236575fbf0c6ae33ab727dd43e130d" gracePeriod=25 Nov 01 09:14:05.361907 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.361860 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" podUID="8a24f164-c457-4d88-bfb6-0e2528630db4" containerName="payload-pre-processing" containerID="cri-o://533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6" gracePeriod=25 Nov 01 09:14:05.473502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.473466 5710 generic.go:356] "Generic (PLEG): container finished" podID="26846be6-9903-4d21-9e70-e32de7fbe366" containerID="596796f7335ca4483347338df87bd4f8a9236575fbf0c6ae33ab727dd43e130d" exitCode=0 Nov 01 09:14:05.473705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.473524 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" event={"ID":"26846be6-9903-4d21-9e70-e32de7fbe366","Type":"ContainerDied","Data":"596796f7335ca4483347338df87bd4f8a9236575fbf0c6ae33ab727dd43e130d"} Nov 01 09:14:05.544528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.544499 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:14:05.633627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.633601 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:14:05.653960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.653927 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume\") pod \"26846be6-9903-4d21-9e70-e32de7fbe366\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " Nov 01 09:14:05.654204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.654188 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw\" (UniqueName: \"kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw\") pod \"26846be6-9903-4d21-9e70-e32de7fbe366\" (UID: \"26846be6-9903-4d21-9e70-e32de7fbe366\") " Nov 01 09:14:05.655073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.655045 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume" pod "26846be6-9903-4d21-9e70-e32de7fbe366" (UID: "26846be6-9903-4d21-9e70-e32de7fbe366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:05.657042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.657015 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw" pod "26846be6-9903-4d21-9e70-e32de7fbe366" (UID: "26846be6-9903-4d21-9e70-e32de7fbe366"). InnerVolumeSpecName "kube-api-access-vv7gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:05.755094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.755055 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw\" (UniqueName: \"kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw\") pod \"8a24f164-c457-4d88-bfb6-0e2528630db4\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " Nov 01 09:14:05.755283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.755134 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume\") pod \"8a24f164-c457-4d88-bfb6-0e2528630db4\" (UID: \"8a24f164-c457-4d88-bfb6-0e2528630db4\") " Nov 01 09:14:05.755517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.755495 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume" pod "8a24f164-c457-4d88-bfb6-0e2528630db4" (UID: "8a24f164-c457-4d88-bfb6-0e2528630db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:05.755792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.755777 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26846be6-9903-4d21-9e70-e32de7fbe366-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:05.755849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.755795 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a24f164-c457-4d88-bfb6-0e2528630db4-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:05.755849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.755805 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7gw\" (UniqueName: \"kubernetes.io/projected/26846be6-9903-4d21-9e70-e32de7fbe366-kube-api-access-vv7gw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:05.757575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.757548 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw" pod "8a24f164-c457-4d88-bfb6-0e2528630db4" (UID: "8a24f164-c457-4d88-bfb6-0e2528630db4"). InnerVolumeSpecName "kube-api-access-l5kzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:05.856805 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:05.856755 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5kzw\" (UniqueName: \"kubernetes.io/projected/8a24f164-c457-4d88-bfb6-0e2528630db4-kube-api-access-l5kzw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:06.486210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.486174 5710 generic.go:356] "Generic (PLEG): container finished" podID="8a24f164-c457-4d88-bfb6-0e2528630db4" containerID="533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6" exitCode=0 Nov 01 09:14:06.486753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.486239 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" Nov 01 09:14:06.486753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.486260 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" event={"ID":"8a24f164-c457-4d88-bfb6-0e2528630db4","Type":"ContainerDied","Data":"533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6"} Nov 01 09:14:06.486753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.486324 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj" event={"ID":"8a24f164-c457-4d88-bfb6-0e2528630db4","Type":"ContainerDied","Data":"bcd07be3be0d5e98a9e4f89898db9e05bf20cb120266f3e2001cfae89f85c07e"} Nov 01 09:14:06.486753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.486354 5710 scope.go:122] "RemoveContainer" containerID="533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6" Nov 01 09:14:06.488224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.488145 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" event={"ID":"26846be6-9903-4d21-9e70-e32de7fbe366","Type":"ContainerDied","Data":"980ef31a4f2930a52cb49723bd74052b393389418335919553b4291a0a7d5b89"} Nov 01 09:14:06.488224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.488240 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5" Nov 01 09:14:06.498249 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.498218 5710 scope.go:122] "RemoveContainer" containerID="533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6" Nov 01 09:14:06.498622 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:06.498599 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6\": container with ID starting with 533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6 not found: ID does not exist" containerID="533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6" Nov 01 09:14:06.498704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.498633 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6"} err="failed to get container status \"533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6\": rpc error: code = NotFound desc = could not find container \"533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6\": container with ID starting with 533bd6ce9f8cdc13b94a70a8d6631b623d23df5eb4c03df62b427ccb3bf8deb6 not found: ID does not exist" Nov 01 09:14:06.498704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.498656 5710 scope.go:122] "RemoveContainer" containerID="596796f7335ca4483347338df87bd4f8a9236575fbf0c6ae33ab727dd43e130d" Nov 01 09:14:06.641335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.641301 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5"] Nov 01 09:14:06.695529 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.695449 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c3aaa7-7fbbb79bd7-gzhr5"] Nov 01 09:14:06.752496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.752458 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj"] Nov 01 09:14:06.756993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:06.756962 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c3aaa7-db8d54c5f-g67fj"] Nov 01 09:14:07.060445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:07.060365 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26846be6-9903-4d21-9e70-e32de7fbe366" path="/var/lib/kubelet/pods/26846be6-9903-4d21-9e70-e32de7fbe366/volumes" Nov 01 09:14:07.060832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:07.060817 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a24f164-c457-4d88-bfb6-0e2528630db4" path="/var/lib/kubelet/pods/8a24f164-c457-4d88-bfb6-0e2528630db4/volumes" Nov 01 09:14:15.478358 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:15.478312 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:14:35.678293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:35.678254 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:14:36.360111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.360071 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg"] Nov 01 09:14:36.360332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.360310 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" podUID="94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" containerName="maas-api" containerID="cri-o://f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d" gracePeriod=30 Nov 01 09:14:36.677083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.677047 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:14:36.721868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.721820 5710 generic.go:356] "Generic (PLEG): container finished" podID="94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" containerID="f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d" exitCode=0 Nov 01 09:14:36.722321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.721882 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" Nov 01 09:14:36.722321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.721894 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" event={"ID":"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1","Type":"ContainerDied","Data":"f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d"} Nov 01 09:14:36.722321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.721952 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg" event={"ID":"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1","Type":"ContainerDied","Data":"135abe4155979252feb02bd277a3f55a000ce46335df30d05f3c5e2b0a2b5bd8"} Nov 01 09:14:36.722321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.721983 5710 scope.go:122] "RemoveContainer" containerID="f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d" Nov 01 09:14:36.732860 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.732837 5710 scope.go:122] "RemoveContainer" containerID="f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d" Nov 01 09:14:36.733273 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:36.733247 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d\": container with ID starting with f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d not found: ID does not exist" containerID="f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d" Nov 01 09:14:36.733373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.733282 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d"} err="failed to get container status \"f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d\": rpc error: code = NotFound desc = could not find container \"f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d\": container with ID starting with f860616d089f6324bd7f63887cf1ec07cde7f71923d3a20de6017fd6e4ebeb0d not found: ID does not exist" Nov 01 09:14:36.803139 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.803024 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs\") pod \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " Nov 01 09:14:36.803431 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.803411 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca\") pod \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " Nov 01 09:14:36.803513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.803450 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\") pod \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " Nov 01 09:14:36.803608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.803592 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b\" (UniqueName: \"kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b\") pod \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\" (UID: \"94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1\") " Nov 01 09:14:36.804216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.804176 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca" pod "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" (UID: "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:36.804336 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.804323 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:36.805839 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.805814 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs" pod "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" (UID: "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:14:36.806503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.806478 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls" pod "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" (UID: "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:14:36.807157 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.807121 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b" pod "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" (UID: "94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1"). InnerVolumeSpecName "kube-api-access-72h8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:36.832961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.832921 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx"] Nov 01 09:14:36.833267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.833237 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" podUID="d72205af-d592-4096-aba0-24efe89c01ee" containerName="payload-processing" containerID="cri-o://10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b" gracePeriod=30 Nov 01 09:14:36.888317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.888280 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4"] Nov 01 09:14:36.888511 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.888475 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" podUID="1bebc555-84e5-4f38-a390-58c1ccf4b027" containerName="payload-pre-processing" containerID="cri-o://332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919" gracePeriod=30 Nov 01 09:14:36.905730 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.905681 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72h8b\" (UniqueName: \"kubernetes.io/projected/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-kube-api-access-72h8b\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:36.905846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.905752 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:36.905846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:36.905771 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:37.149229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.149197 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:14:37.185230 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.185201 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:14:37.186828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.186712 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg"] Nov 01 09:14:37.192216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.192174 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b893c8-6f966f4985-6r5vg"] Nov 01 09:14:37.209751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.209677 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume\") pod \"d72205af-d592-4096-aba0-24efe89c01ee\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " Nov 01 09:14:37.209936 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.209852 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq\" (UniqueName: \"kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq\") pod \"d72205af-d592-4096-aba0-24efe89c01ee\" (UID: \"d72205af-d592-4096-aba0-24efe89c01ee\") " Nov 01 09:14:37.210159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.210124 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume" pod "d72205af-d592-4096-aba0-24efe89c01ee" (UID: "d72205af-d592-4096-aba0-24efe89c01ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:37.210458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.210432 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72205af-d592-4096-aba0-24efe89c01ee-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:37.212349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.212329 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq" pod "d72205af-d592-4096-aba0-24efe89c01ee" (UID: "d72205af-d592-4096-aba0-24efe89c01ee"). InnerVolumeSpecName "kube-api-access-fqwwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:37.311860 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.311827 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t\" (UniqueName: \"kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t\") pod \"1bebc555-84e5-4f38-a390-58c1ccf4b027\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " Nov 01 09:14:37.312071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.311912 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume\") pod \"1bebc555-84e5-4f38-a390-58c1ccf4b027\" (UID: \"1bebc555-84e5-4f38-a390-58c1ccf4b027\") " Nov 01 09:14:37.312236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.312207 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqwwq\" (UniqueName: \"kubernetes.io/projected/d72205af-d592-4096-aba0-24efe89c01ee-kube-api-access-fqwwq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:37.312355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.312322 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume" pod "1bebc555-84e5-4f38-a390-58c1ccf4b027" (UID: "1bebc555-84e5-4f38-a390-58c1ccf4b027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:37.314418 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.314384 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t" pod "1bebc555-84e5-4f38-a390-58c1ccf4b027" (UID: "1bebc555-84e5-4f38-a390-58c1ccf4b027"). InnerVolumeSpecName "kube-api-access-h8t2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:37.414022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.413612 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8t2t\" (UniqueName: \"kubernetes.io/projected/1bebc555-84e5-4f38-a390-58c1ccf4b027-kube-api-access-h8t2t\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:37.414022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.413653 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bebc555-84e5-4f38-a390-58c1ccf4b027-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:37.730415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.730375 5710 generic.go:356] "Generic (PLEG): container finished" podID="1bebc555-84e5-4f38-a390-58c1ccf4b027" containerID="332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919" exitCode=0 Nov 01 09:14:37.730947 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.730422 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" Nov 01 09:14:37.730947 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.730454 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" event={"ID":"1bebc555-84e5-4f38-a390-58c1ccf4b027","Type":"ContainerDied","Data":"332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919"} Nov 01 09:14:37.730947 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.730510 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4" event={"ID":"1bebc555-84e5-4f38-a390-58c1ccf4b027","Type":"ContainerDied","Data":"81e9bd5ecfe5a19e83839f58de95bc8d4fae70d9b0386c849107a6341809d5f6"} Nov 01 09:14:37.730947 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.730542 5710 scope.go:122] "RemoveContainer" containerID="332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919" Nov 01 09:14:37.732160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.732126 5710 generic.go:356] "Generic (PLEG): container finished" podID="d72205af-d592-4096-aba0-24efe89c01ee" containerID="10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b" exitCode=0 Nov 01 09:14:37.732305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.732168 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" event={"ID":"d72205af-d592-4096-aba0-24efe89c01ee","Type":"ContainerDied","Data":"10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b"} Nov 01 09:14:37.732305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.732217 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" event={"ID":"d72205af-d592-4096-aba0-24efe89c01ee","Type":"ContainerDied","Data":"4cf99ffb9857e91bb0721bd17162961579a0f77c0c49948fb7bc68f2aae21633"} Nov 01 09:14:37.732305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.732176 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx" Nov 01 09:14:37.743871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.743847 5710 scope.go:122] "RemoveContainer" containerID="332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919" Nov 01 09:14:37.744238 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:37.744215 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919\": container with ID starting with 332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919 not found: ID does not exist" containerID="332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919" Nov 01 09:14:37.744334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.744250 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919"} err="failed to get container status \"332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919\": rpc error: code = NotFound desc = could not find container \"332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919\": container with ID starting with 332003a2f890ac2c609705ebbd58e4ddd7a1b4acc26282352b5983f4d0206919 not found: ID does not exist" Nov 01 09:14:37.744334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.744277 5710 scope.go:122] "RemoveContainer" containerID="10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b" Nov 01 09:14:37.753920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.753895 5710 scope.go:122] "RemoveContainer" containerID="10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b" Nov 01 09:14:37.754286 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:37.754265 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b\": container with ID starting with 10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b not found: ID does not exist" containerID="10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b" Nov 01 09:14:37.754354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.754298 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b"} err="failed to get container status \"10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b\": rpc error: code = NotFound desc = could not find container \"10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b\": container with ID starting with 10ae61ec3d26fa93141645affc2e9b38fe9f2b9fc977c3e8ca293b8d7abac77b not found: ID does not exist" Nov 01 09:14:37.898671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.898633 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4"] Nov 01 09:14:37.903111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:37.903079 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b893c8-6bb8f8887f-fbzb4"] Nov 01 09:14:38.010548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:38.009904 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx"] Nov 01 09:14:38.013594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:38.013481 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b893c8-d7c5ccb76-jd9nx"] Nov 01 09:14:38.723354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:38.723310 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp"] Nov 01 09:14:38.723607 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:38.723561 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" containerID="cri-o://ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d" gracePeriod=30 Nov 01 09:14:39.059008 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:39.058925 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bebc555-84e5-4f38-a390-58c1ccf4b027" path="/var/lib/kubelet/pods/1bebc555-84e5-4f38-a390-58c1ccf4b027/volumes" Nov 01 09:14:39.059437 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:39.059418 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" path="/var/lib/kubelet/pods/94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1/volumes" Nov 01 09:14:39.059835 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:39.059820 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72205af-d592-4096-aba0-24efe89c01ee" path="/var/lib/kubelet/pods/d72205af-d592-4096-aba0-24efe89c01ee/volumes" Nov 01 09:14:44.041567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.041539 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:14:44.114564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114519 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.114851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114583 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.114851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114603 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.114851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114622 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.114851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114752 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.114851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114791 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.114851 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114843 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.115189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114916 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.115189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114951 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:44.115189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114969 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token\") pod \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\" (UID: \"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3\") " Nov 01 09:14:44.115189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.114991 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:44.115189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115083 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:44.115455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115244 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:44.115455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115275 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:44.115595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115520 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.115595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115544 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.115595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115559 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.115948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115650 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.115948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.115762 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.117341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.117309 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:14:44.117467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.117339 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:44.117467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.117389 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "kube-api-access-hw4m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:44.117467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.117412 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token" pod "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" (UID: "88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:44.217038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.217004 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4m8\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-kube-api-access-hw4m8\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.217038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.217033 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.217038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.217043 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.217267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.217052 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:44.782427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.782387 5710 generic.go:356] "Generic (PLEG): container finished" podID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerID="ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d" exitCode=0 Nov 01 09:14:44.782628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.782444 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" Nov 01 09:14:44.782628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.782463 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" event={"ID":"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3","Type":"ContainerDied","Data":"ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d"} Nov 01 09:14:44.782628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.782520 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp" event={"ID":"88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3","Type":"ContainerDied","Data":"6bf978ea7cf9631517bab0347ae7c44925dcb2a70ea3cea2b1a8bcbbf90ea5e4"} Nov 01 09:14:44.782628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.782541 5710 scope.go:122] "RemoveContainer" containerID="ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d" Nov 01 09:14:44.793299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.793273 5710 scope.go:122] "RemoveContainer" containerID="629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea" Nov 01 09:14:44.805022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.804999 5710 scope.go:122] "RemoveContainer" containerID="ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d" Nov 01 09:14:44.805343 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:44.805322 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d\": container with ID starting with ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d not found: ID does not exist" containerID="ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d" Nov 01 09:14:44.805397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.805351 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d"} err="failed to get container status \"ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d\": rpc error: code = NotFound desc = could not find container \"ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d\": container with ID starting with ffd6cecdb0a41a4ffde92f285894fd2481b65d61cbeef1f2d1c731e804a0a44d not found: ID does not exist" Nov 01 09:14:44.805397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.805368 5710 scope.go:122] "RemoveContainer" containerID="629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea" Nov 01 09:14:44.805619 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:44.805599 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea\": container with ID starting with 629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea not found: ID does not exist" containerID="629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea" Nov 01 09:14:44.805666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.805630 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea"} err="failed to get container status \"629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea\": rpc error: code = NotFound desc = could not find container \"629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea\": container with ID starting with 629f9ed429083a26b687b7d8ba1ba516321aeb2f5e7597ab356bc0d05b6952ea not found: ID does not exist" Nov 01 09:14:44.936758 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.936692 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp"] Nov 01 09:14:44.940330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:44.940295 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b893c8-openshift-default-dcd9c8845-csbpp"] Nov 01 09:14:45.059091 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:45.059010 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" path="/var/lib/kubelet/pods/88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3/volumes" Nov 01 09:14:45.480925 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:45.480884 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:14:49.408509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:49.408473 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww"] Nov 01 09:14:49.409019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:49.408700 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" containerID="cri-o://451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842" gracePeriod=30 Nov 01 09:14:54.685385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.685312 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:14:54.744806 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744768 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.744994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744850 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.744994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744876 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.744994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744903 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.744994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744931 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.744994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744955 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.745371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.744998 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.745371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745078 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.745371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745118 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data\") pod \"63fd7f11-48e2-4703-af42-22d29b0cd125\" (UID: \"63fd7f11-48e2-4703-af42-22d29b0cd125\") " Nov 01 09:14:54.745371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745117 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:54.745371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745240 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:54.745654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745378 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:54.745654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745515 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:54.745654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745597 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:54.746029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745707 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.746029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745750 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.746029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745766 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.746029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.745779 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.747578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.747552 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "kube-api-access-mxtvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:54.747578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.747565 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:54.747850 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.747806 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:14:54.747894 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.747840 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo" pod "63fd7f11-48e2-4703-af42-22d29b0cd125" (UID: "63fd7f11-48e2-4703-af42-22d29b0cd125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:14:54.846676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.846643 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.846676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.846671 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.846676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.846682 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fd7f11-48e2-4703-af42-22d29b0cd125-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.847031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.846693 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtvl\" (UniqueName: \"kubernetes.io/projected/63fd7f11-48e2-4703-af42-22d29b0cd125-kube-api-access-mxtvl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.847031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.846706 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fd7f11-48e2-4703-af42-22d29b0cd125-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:54.850000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.849965 5710 generic.go:356] "Generic (PLEG): container finished" podID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerID="451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842" exitCode=0 Nov 01 09:14:54.850194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.850032 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" Nov 01 09:14:54.850194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.850061 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" event={"ID":"63fd7f11-48e2-4703-af42-22d29b0cd125","Type":"ContainerDied","Data":"451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842"} Nov 01 09:14:54.850194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.850154 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww" event={"ID":"63fd7f11-48e2-4703-af42-22d29b0cd125","Type":"ContainerDied","Data":"72899e73785fa66ea083eb7296383aeb0768e889de6f71424a564602f4d36125"} Nov 01 09:14:54.850381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.850197 5710 scope.go:122] "RemoveContainer" containerID="451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842" Nov 01 09:14:54.862155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.862125 5710 scope.go:122] "RemoveContainer" containerID="e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202" Nov 01 09:14:54.873793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.873768 5710 scope.go:122] "RemoveContainer" containerID="451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842" Nov 01 09:14:54.874087 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:54.874062 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842\": container with ID starting with 451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842 not found: ID does not exist" containerID="451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842" Nov 01 09:14:54.874170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.874094 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842"} err="failed to get container status \"451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842\": rpc error: code = NotFound desc = could not find container \"451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842\": container with ID starting with 451478fae17b944efbc4d1841197e45f86d6f949c4c730d3647fed24dfd24842 not found: ID does not exist" Nov 01 09:14:54.874170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.874113 5710 scope.go:122] "RemoveContainer" containerID="e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202" Nov 01 09:14:54.874377 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:14:54.874359 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202\": container with ID starting with e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202 not found: ID does not exist" containerID="e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202" Nov 01 09:14:54.874485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:54.874383 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202"} err="failed to get container status \"e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202\": rpc error: code = NotFound desc = could not find container \"e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202\": container with ID starting with e888f5306c80400f9890f33d8c3daa93123c2f12e6cddac6a59444605b790202 not found: ID does not exist" Nov 01 09:14:55.005397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:55.005307 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww"] Nov 01 09:14:55.011443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:55.011411 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c3aaa7-openshift-default-5974f64565-djhww"] Nov 01 09:14:55.058979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:55.058946 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" path="/var/lib/kubelet/pods/63fd7f11-48e2-4703-af42-22d29b0cd125/volumes" Nov 01 09:14:55.361789 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:55.361756 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc"] Nov 01 09:14:55.459435 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:55.459396 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb"] Nov 01 09:14:56.626290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.626247 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv"] Nov 01 09:14:56.626874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.626477 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" podUID="b04181e8-dffe-4c5d-b894-2e1a74914510" containerName="maas-api" containerID="cri-o://ffdfb0f942987affc0d9bc84d5bf30546ad72b6dea7ad499599f86b58f14c7e2" gracePeriod=30 Nov 01 09:14:56.866640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.866598 5710 generic.go:356] "Generic (PLEG): container finished" podID="b04181e8-dffe-4c5d-b894-2e1a74914510" containerID="ffdfb0f942987affc0d9bc84d5bf30546ad72b6dea7ad499599f86b58f14c7e2" exitCode=0 Nov 01 09:14:56.866864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.866640 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" event={"ID":"b04181e8-dffe-4c5d-b894-2e1a74914510","Type":"ContainerDied","Data":"ffdfb0f942987affc0d9bc84d5bf30546ad72b6dea7ad499599f86b58f14c7e2"} Nov 01 09:14:56.923567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.923534 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:14:56.977153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.976239 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca\") pod \"b04181e8-dffe-4c5d-b894-2e1a74914510\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " Nov 01 09:14:56.977153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.976291 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\") pod \"b04181e8-dffe-4c5d-b894-2e1a74914510\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " Nov 01 09:14:56.977153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.976339 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs\") pod \"b04181e8-dffe-4c5d-b894-2e1a74914510\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " Nov 01 09:14:56.977153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.976377 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl\" (UniqueName: \"kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl\") pod \"b04181e8-dffe-4c5d-b894-2e1a74914510\" (UID: \"b04181e8-dffe-4c5d-b894-2e1a74914510\") " Nov 01 09:14:56.977827 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.977795 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca" pod "b04181e8-dffe-4c5d-b894-2e1a74914510" (UID: "b04181e8-dffe-4c5d-b894-2e1a74914510"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:14:56.998746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.983174 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl" pod "b04181e8-dffe-4c5d-b894-2e1a74914510" (UID: "b04181e8-dffe-4c5d-b894-2e1a74914510"). InnerVolumeSpecName "kube-api-access-4v6bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:14:56.999338 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:56.999310 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs" pod "b04181e8-dffe-4c5d-b894-2e1a74914510" (UID: "b04181e8-dffe-4c5d-b894-2e1a74914510"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:14:57.007092 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.007051 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls" pod "b04181e8-dffe-4c5d-b894-2e1a74914510" (UID: "b04181e8-dffe-4c5d-b894-2e1a74914510"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:14:57.079166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.079132 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04181e8-dffe-4c5d-b894-2e1a74914510-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:57.079331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.079171 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:57.079331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.079186 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04181e8-dffe-4c5d-b894-2e1a74914510-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:57.079331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.079200 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6bl\" (UniqueName: \"kubernetes.io/projected/b04181e8-dffe-4c5d-b894-2e1a74914510-kube-api-access-4v6bl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:14:57.874874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.874831 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" event={"ID":"b04181e8-dffe-4c5d-b894-2e1a74914510","Type":"ContainerDied","Data":"884f84fd9c82ac4fb25666fbb69c8bae6cd91fb6f084d8abb31fa1f1e660a425"} Nov 01 09:14:57.875404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.874900 5710 scope.go:122] "RemoveContainer" containerID="ffdfb0f942987affc0d9bc84d5bf30546ad72b6dea7ad499599f86b58f14c7e2" Nov 01 09:14:57.875404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:57.874847 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv" Nov 01 09:14:58.029768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:58.029661 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv"] Nov 01 09:14:58.075665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:58.075633 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9f4e9b-7fd9f655f8-r4psv"] Nov 01 09:14:59.058828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:14:59.058785 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04181e8-dffe-4c5d-b894-2e1a74914510" path="/var/lib/kubelet/pods/b04181e8-dffe-4c5d-b894-2e1a74914510/volumes" Nov 01 09:15:00.167214 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167160 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq"] Nov 01 09:15:00.167842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167824 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167843 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167854 5710 cpu_manager.go:414] "containerMap: removing container" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167860 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167866 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b04181e8-dffe-4c5d-b894-2e1a74914510" containerName="maas-api" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167872 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04181e8-dffe-4c5d-b894-2e1a74914510" containerName="maas-api" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167893 5710 cpu_manager.go:414] "containerMap: removing container" podUID="93935dae-3a7a-4ff3-a4fb-f193691b4147" containerName="maas-api" Nov 01 09:15:00.167912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167898 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93935dae-3a7a-4ff3-a4fb-f193691b4147" containerName="maas-api" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167955 5710 cpu_manager.go:414] "containerMap: removing container" podUID="26846be6-9903-4d21-9e70-e32de7fbe366" containerName="payload-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167965 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26846be6-9903-4d21-9e70-e32de7fbe366" containerName="payload-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167989 5710 cpu_manager.go:414] "containerMap: removing container" podUID="94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" containerName="maas-api" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.167996 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" containerName="maas-api" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168008 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1bebc555-84e5-4f38-a390-58c1ccf4b027" containerName="payload-pre-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168015 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bebc555-84e5-4f38-a390-58c1ccf4b027" containerName="payload-pre-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168028 5710 cpu_manager.go:414] "containerMap: removing container" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168035 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168046 5710 cpu_manager.go:414] "containerMap: removing container" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168053 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168080 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168085 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168091 5710 cpu_manager.go:414] "containerMap: removing container" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168097 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168113 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d72205af-d592-4096-aba0-24efe89c01ee" containerName="payload-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168119 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72205af-d592-4096-aba0-24efe89c01ee" containerName="payload-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168129 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8a24f164-c457-4d88-bfb6-0e2528630db4" containerName="payload-pre-processing" Nov 01 09:15:00.168137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168134 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a24f164-c457-4d88-bfb6-0e2528630db4" containerName="payload-pre-processing" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168156 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168161 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168321 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="93935dae-3a7a-4ff3-a4fb-f193691b4147" containerName="maas-api" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168341 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="26846be6-9903-4d21-9e70-e32de7fbe366" containerName="payload-processing" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168353 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f4b186-c5eb-4d00-b9b8-c7a7bd63dfb1" containerName="maas-api" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168362 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bebc555-84e5-4f38-a390-58c1ccf4b027" containerName="payload-pre-processing" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168369 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168374 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fd7f11-48e2-4703-af42-22d29b0cd125" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168382 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168387 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168393 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72205af-d592-4096-aba0-24efe89c01ee" containerName="payload-processing" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168403 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a24f164-c457-4d88-bfb6-0e2528630db4" containerName="payload-pre-processing" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168416 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31ff5cf-7c02-470b-b693-6a426ef895ec" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168425 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4eb97f-3177-47da-befe-2cde7b743657" containerName="istio-proxy" Nov 01 09:15:00.168675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.168431 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04181e8-dffe-4c5d-b894-2e1a74914510" containerName="maas-api" Nov 01 09:15:00.173926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.173900 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.184181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.184153 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9"] Nov 01 09:15:00.184973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.184952 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="88eaedd8-27f5-4f7a-baf4-39c7f12ed3b3" containerName="istio-proxy" Nov 01 09:15:00.188310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.188290 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.194540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.194514 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4"] Nov 01 09:15:00.199128 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.199100 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.213570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.213493 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh"] Nov 01 09:15:00.217795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.217775 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.220528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.220504 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.220676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.220636 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhl8\" (UniqueName: \"kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.220791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.220773 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.220924 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.220909 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7l7q\" (UniqueName: \"kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.228369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.228341 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\"" Nov 01 09:15:00.267031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.266993 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq"] Nov 01 09:15:00.312514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.312473 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9"] Nov 01 09:15:00.315191 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.315148 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4"] Nov 01 09:15:00.317414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.317387 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh"] Nov 01 09:15:00.321873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.321846 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.322048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.321886 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzn6k\" (UniqueName: \"kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k\") pod \"maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.322048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.321915 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.322171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.322095 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7l7q\" (UniqueName: \"kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.322171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.322129 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rlw\" (UniqueName: \"kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw\") pod \"maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.322171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.322156 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.322342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.322254 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.322342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.322296 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhl8\" (UniqueName: \"kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.322641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.322620 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.323027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.323002 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.362986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.362938 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" podUID="691b3adb-9c90-4da5-9327-0351757a7dcd" containerName="payload-processing" containerID="cri-o://389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d" gracePeriod=25 Nov 01 09:15:00.380376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.380348 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7l7q\" (UniqueName: \"kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q\") pod \"maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.380376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.380365 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhl8\" (UniqueName: \"kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.423367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.423326 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzn6k\" (UniqueName: \"kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k\") pod \"maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.423367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.423379 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.423618 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.423431 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rlw\" (UniqueName: \"kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw\") pod \"maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.423618 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.423487 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.424136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.424114 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.424205 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.424135 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.460313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.460269 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" podUID="af4645f8-6919-4682-ad2d-738ad3dd6dae" containerName="payload-pre-processing" containerID="cri-o://06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd" gracePeriod=25 Nov 01 09:15:00.482869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.482839 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rlw\" (UniqueName: \"kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw\") pod \"maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.482986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.482839 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzn6k\" (UniqueName: \"kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k\") pod \"maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.488546 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.488519 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:00.502612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.502590 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:00.514883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.514859 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:00.532776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.532744 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:00.774303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.774264 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:15:00.824764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.824734 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:15:00.827187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.827161 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc\" (UniqueName: \"kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc\") pod \"691b3adb-9c90-4da5-9327-0351757a7dcd\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " Nov 01 09:15:00.827310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.827239 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume\") pod \"691b3adb-9c90-4da5-9327-0351757a7dcd\" (UID: \"691b3adb-9c90-4da5-9327-0351757a7dcd\") " Nov 01 09:15:00.827576 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.827552 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume" pod "691b3adb-9c90-4da5-9327-0351757a7dcd" (UID: "691b3adb-9c90-4da5-9327-0351757a7dcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:00.827773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.827755 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691b3adb-9c90-4da5-9327-0351757a7dcd-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:00.829387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.829367 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc" pod "691b3adb-9c90-4da5-9327-0351757a7dcd" (UID: "691b3adb-9c90-4da5-9327-0351757a7dcd"). InnerVolumeSpecName "kube-api-access-j7tmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:00.874757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.874702 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq"] Nov 01 09:15:00.881927 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:00.881893 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d04a58_0cb7_48b0_a1ec_0d78c06601ba.slice/crio-1ca030e60dcb6f4cf0e7ea290545a4ad117466c1081d30ac3ff6df7b81fecf7e WatchSource:0}: Error finding container 1ca030e60dcb6f4cf0e7ea290545a4ad117466c1081d30ac3ff6df7b81fecf7e: Status 404 returned error can't find the container with id 1ca030e60dcb6f4cf0e7ea290545a4ad117466c1081d30ac3ff6df7b81fecf7e Nov 01 09:15:00.907732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.907677 5710 generic.go:356] "Generic (PLEG): container finished" podID="af4645f8-6919-4682-ad2d-738ad3dd6dae" containerID="06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd" exitCode=0 Nov 01 09:15:00.907883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.907800 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" Nov 01 09:15:00.907883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.907817 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" event={"ID":"af4645f8-6919-4682-ad2d-738ad3dd6dae","Type":"ContainerDied","Data":"06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd"} Nov 01 09:15:00.908027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.907883 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb" event={"ID":"af4645f8-6919-4682-ad2d-738ad3dd6dae","Type":"ContainerDied","Data":"cc0006132beb467380ac3fa943df05fc18661520e6142bfc70f939622cdb8e6a"} Nov 01 09:15:00.908027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.907920 5710 scope.go:122] "RemoveContainer" containerID="06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd" Nov 01 09:15:00.909487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.909459 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" event={"ID":"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba","Type":"ContainerStarted","Data":"1ca030e60dcb6f4cf0e7ea290545a4ad117466c1081d30ac3ff6df7b81fecf7e"} Nov 01 09:15:00.911076 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.911050 5710 generic.go:356] "Generic (PLEG): container finished" podID="691b3adb-9c90-4da5-9327-0351757a7dcd" containerID="389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d" exitCode=0 Nov 01 09:15:00.911234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.911085 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" event={"ID":"691b3adb-9c90-4da5-9327-0351757a7dcd","Type":"ContainerDied","Data":"389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d"} Nov 01 09:15:00.911234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.911100 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" Nov 01 09:15:00.911234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.911127 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc" event={"ID":"691b3adb-9c90-4da5-9327-0351757a7dcd","Type":"ContainerDied","Data":"1746e876fa35b220918b9b336cd6c822df1b2fa97709c8e118fcaa7d0ce2d72b"} Nov 01 09:15:00.929885 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.929853 5710 scope.go:122] "RemoveContainer" containerID="06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd" Nov 01 09:15:00.930667 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:00.930637 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd\": container with ID starting with 06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd not found: ID does not exist" containerID="06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd" Nov 01 09:15:00.930838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.930681 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd"} err="failed to get container status \"06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd\": rpc error: code = NotFound desc = could not find container \"06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd\": container with ID starting with 06c4baf1904c378d150cf6236295085f8ffdb0b009873daf25303edcaf70a8dd not found: ID does not exist" Nov 01 09:15:00.930838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.930708 5710 scope.go:122] "RemoveContainer" containerID="389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d" Nov 01 09:15:00.931336 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.931315 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume\") pod \"af4645f8-6919-4682-ad2d-738ad3dd6dae\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " Nov 01 09:15:00.931426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.931407 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62\" (UniqueName: \"kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62\") pod \"af4645f8-6919-4682-ad2d-738ad3dd6dae\" (UID: \"af4645f8-6919-4682-ad2d-738ad3dd6dae\") " Nov 01 09:15:00.932891 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.932860 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7tmc\" (UniqueName: \"kubernetes.io/projected/691b3adb-9c90-4da5-9327-0351757a7dcd-kube-api-access-j7tmc\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:00.934823 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.933908 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume" pod "af4645f8-6919-4682-ad2d-738ad3dd6dae" (UID: "af4645f8-6919-4682-ad2d-738ad3dd6dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:00.935161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.935045 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9"] Nov 01 09:15:00.937014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.936991 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62" pod "af4645f8-6919-4682-ad2d-738ad3dd6dae" (UID: "af4645f8-6919-4682-ad2d-738ad3dd6dae"). InnerVolumeSpecName "kube-api-access-zkj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:00.944965 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:00.940294 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc689078_654a_4be1_8dc6_ad49900f4582.slice/crio-7989b8ebdf6c75323cd081255e92233d682ab7435d46285a44c6d552836ffe10 WatchSource:0}: Error finding container 7989b8ebdf6c75323cd081255e92233d682ab7435d46285a44c6d552836ffe10: Status 404 returned error can't find the container with id 7989b8ebdf6c75323cd081255e92233d682ab7435d46285a44c6d552836ffe10 Nov 01 09:15:00.949989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.949952 5710 scope.go:122] "RemoveContainer" containerID="389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d" Nov 01 09:15:00.950464 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:00.950434 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d\": container with ID starting with 389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d not found: ID does not exist" containerID="389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d" Nov 01 09:15:00.950596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.950476 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d"} err="failed to get container status \"389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d\": rpc error: code = NotFound desc = could not find container \"389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d\": container with ID starting with 389a6104be5b64d014eb755f7be177cf82e339df1cb7382c356f5b245b0d474d not found: ID does not exist" Nov 01 09:15:00.953767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.953710 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4"] Nov 01 09:15:00.960854 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:00.960823 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa0cd55_b3eb_422f_87d0_f2af49632018.slice/crio-058d407abbcf7045a33cf213e6a33e141bd7cc2aa740db2f693aec87506a5f9f WatchSource:0}: Error finding container 058d407abbcf7045a33cf213e6a33e141bd7cc2aa740db2f693aec87506a5f9f: Status 404 returned error can't find the container with id 058d407abbcf7045a33cf213e6a33e141bd7cc2aa740db2f693aec87506a5f9f Nov 01 09:15:00.980216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:00.980191 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh"] Nov 01 09:15:00.988888 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:00.988859 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e365490_d0bf_4598_bd85_7fa0f33cf8cb.slice/crio-a8e08a442f62c7d31f6e3c3b02f65cd53fdd54e635558d51916ac91cace69a5e WatchSource:0}: Error finding container a8e08a442f62c7d31f6e3c3b02f65cd53fdd54e635558d51916ac91cace69a5e: Status 404 returned error can't find the container with id a8e08a442f62c7d31f6e3c3b02f65cd53fdd54e635558d51916ac91cace69a5e Nov 01 09:15:01.034129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.034096 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4645f8-6919-4682-ad2d-738ad3dd6dae-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:01.034129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.034126 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkj62\" (UniqueName: \"kubernetes.io/projected/af4645f8-6919-4682-ad2d-738ad3dd6dae-kube-api-access-zkj62\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:01.151412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.151366 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc"] Nov 01 09:15:01.156194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.156162 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-55644e-7d7bb646c5-h7hjc"] Nov 01 09:15:01.315439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.315407 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb"] Nov 01 09:15:01.319384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.319353 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-55644e-545f7c584c-pmlxb"] Nov 01 09:15:01.919019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.918980 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerStarted","Data":"316e8fbb32374916382bafdd9b1940ce5b669fcf17772fc0c1e1bc3bab17526f"} Nov 01 09:15:01.919185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.919039 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerStarted","Data":"a8e08a442f62c7d31f6e3c3b02f65cd53fdd54e635558d51916ac91cace69a5e"} Nov 01 09:15:01.921688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.921661 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerStarted","Data":"8e6c76312791b31f89228ddfe0beefa85640a22221cd786f2b4ab9df2bb2cba9"} Nov 01 09:15:01.921864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.921704 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerStarted","Data":"7989b8ebdf6c75323cd081255e92233d682ab7435d46285a44c6d552836ffe10"} Nov 01 09:15:01.923313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.923290 5710 generic.go:356] "Generic (PLEG): container finished" podID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerID="c5f9419e27741151dd4c971cf3f318fa19c9219dfbecf629db205f2a8a48d037" exitCode=60 Nov 01 09:15:01.923451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.923369 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" event={"ID":"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba","Type":"ContainerDied","Data":"c5f9419e27741151dd4c971cf3f318fa19c9219dfbecf629db205f2a8a48d037"} Nov 01 09:15:01.923639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.923617 5710 scope.go:122] "RemoveContainer" containerID="c5f9419e27741151dd4c971cf3f318fa19c9219dfbecf629db205f2a8a48d037" Nov 01 09:15:01.925501 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.925474 5710 generic.go:356] "Generic (PLEG): container finished" podID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerID="be649c0e18e486479b09927bfcab7d17d110efb197f9f10b2f9c42732cb76e5d" exitCode=60 Nov 01 09:15:01.925614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.925530 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" event={"ID":"5aa0cd55-b3eb-422f-87d0-f2af49632018","Type":"ContainerDied","Data":"be649c0e18e486479b09927bfcab7d17d110efb197f9f10b2f9c42732cb76e5d"} Nov 01 09:15:01.925614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.925579 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" event={"ID":"5aa0cd55-b3eb-422f-87d0-f2af49632018","Type":"ContainerStarted","Data":"058d407abbcf7045a33cf213e6a33e141bd7cc2aa740db2f693aec87506a5f9f"} Nov 01 09:15:01.925822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:01.925798 5710 scope.go:122] "RemoveContainer" containerID="be649c0e18e486479b09927bfcab7d17d110efb197f9f10b2f9c42732cb76e5d" Nov 01 09:15:02.028408 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.028354 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" podStartSLOduration=2.028337714 podStartE2EDuration="2.028337714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:02.026787034 +0000 UTC m=+2525.115123719" watchObservedRunningTime="2026-11-01 09:15:02.028337714 +0000 UTC m=+2525.116674386" Nov 01 09:15:02.349576 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.349515 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" podStartSLOduration=2.349492887 podStartE2EDuration="2.349492887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:02.347892776 +0000 UTC m=+2525.436229449" watchObservedRunningTime="2026-11-01 09:15:02.349492887 +0000 UTC m=+2525.437829558" Nov 01 09:15:02.941889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.941838 5710 generic.go:356] "Generic (PLEG): container finished" podID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerID="316e8fbb32374916382bafdd9b1940ce5b669fcf17772fc0c1e1bc3bab17526f" exitCode=60 Nov 01 09:15:02.943106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.942169 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerDied","Data":"316e8fbb32374916382bafdd9b1940ce5b669fcf17772fc0c1e1bc3bab17526f"} Nov 01 09:15:02.943355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.942619 5710 scope.go:122] "RemoveContainer" containerID="316e8fbb32374916382bafdd9b1940ce5b669fcf17772fc0c1e1bc3bab17526f" Nov 01 09:15:02.952423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.952391 5710 generic.go:356] "Generic (PLEG): container finished" podID="dc689078-654a-4be1-8dc6-ad49900f4582" containerID="8e6c76312791b31f89228ddfe0beefa85640a22221cd786f2b4ab9df2bb2cba9" exitCode=60 Nov 01 09:15:02.952589 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.952423 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerDied","Data":"8e6c76312791b31f89228ddfe0beefa85640a22221cd786f2b4ab9df2bb2cba9"} Nov 01 09:15:02.952926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.952798 5710 scope.go:122] "RemoveContainer" containerID="8e6c76312791b31f89228ddfe0beefa85640a22221cd786f2b4ab9df2bb2cba9" Nov 01 09:15:02.955053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.955023 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" event={"ID":"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba","Type":"ContainerStarted","Data":"8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208"} Nov 01 09:15:02.957639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:02.957614 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" event={"ID":"5aa0cd55-b3eb-422f-87d0-f2af49632018","Type":"ContainerStarted","Data":"a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d"} Nov 01 09:15:03.058403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.058366 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691b3adb-9c90-4da5-9327-0351757a7dcd" path="/var/lib/kubelet/pods/691b3adb-9c90-4da5-9327-0351757a7dcd/volumes" Nov 01 09:15:03.058831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.058818 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4645f8-6919-4682-ad2d-738ad3dd6dae" path="/var/lib/kubelet/pods/af4645f8-6919-4682-ad2d-738ad3dd6dae/volumes" Nov 01 09:15:03.158540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.158478 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" podStartSLOduration=3.158460723 podStartE2EDuration="3.158460723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:03.157354469 +0000 UTC m=+2526.245691142" watchObservedRunningTime="2026-11-01 09:15:03.158460723 +0000 UTC m=+2526.246797394" Nov 01 09:15:03.265283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.265172 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" podStartSLOduration=3.265155509 podStartE2EDuration="3.265155509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:03.264595375 +0000 UTC m=+2526.352932048" watchObservedRunningTime="2026-11-01 09:15:03.265155509 +0000 UTC m=+2526.353492178" Nov 01 09:15:03.965970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.965932 5710 generic.go:356] "Generic (PLEG): container finished" podID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerID="8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208" exitCode=60 Nov 01 09:15:03.966429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.966018 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" event={"ID":"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba","Type":"ContainerDied","Data":"8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208"} Nov 01 09:15:03.966429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.966082 5710 scope.go:122] "RemoveContainer" containerID="c5f9419e27741151dd4c971cf3f318fa19c9219dfbecf629db205f2a8a48d037" Nov 01 09:15:03.966429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.966374 5710 scope.go:122] "RemoveContainer" containerID="8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208" Nov 01 09:15:03.966676 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:03.966654 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq_odh-ai-gateway-infra(f6d04a58-0cb7-48b0-a1ec-0d78c06601ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" Nov 01 09:15:03.968364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.968339 5710 generic.go:356] "Generic (PLEG): container finished" podID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerID="a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d" exitCode=60 Nov 01 09:15:03.968497 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.968420 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" event={"ID":"5aa0cd55-b3eb-422f-87d0-f2af49632018","Type":"ContainerDied","Data":"a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d"} Nov 01 09:15:03.968663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.968646 5710 scope.go:122] "RemoveContainer" containerID="a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d" Nov 01 09:15:03.968986 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:03.968966 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4_odh-ai-gateway-infra(5aa0cd55-b3eb-422f-87d0-f2af49632018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" Nov 01 09:15:03.970533 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.970503 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerStarted","Data":"85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d"} Nov 01 09:15:03.972548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.972526 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerStarted","Data":"805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797"} Nov 01 09:15:03.980002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:03.979976 5710 scope.go:122] "RemoveContainer" containerID="be649c0e18e486479b09927bfcab7d17d110efb197f9f10b2f9c42732cb76e5d" Nov 01 09:15:04.980896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.980862 5710 generic.go:356] "Generic (PLEG): container finished" podID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerID="85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d" exitCode=60 Nov 01 09:15:04.981438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.980951 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerDied","Data":"85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d"} Nov 01 09:15:04.981438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.981019 5710 scope.go:122] "RemoveContainer" containerID="316e8fbb32374916382bafdd9b1940ce5b669fcf17772fc0c1e1bc3bab17526f" Nov 01 09:15:04.981438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.981350 5710 scope.go:122] "RemoveContainer" containerID="85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d" Nov 01 09:15:04.981762 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:04.981708 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh_odh-ai-gateway-infra(9e365490-d0bf-4598-bd85-7fa0f33cf8cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" Nov 01 09:15:04.983129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.983105 5710 generic.go:356] "Generic (PLEG): container finished" podID="dc689078-654a-4be1-8dc6-ad49900f4582" containerID="805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797" exitCode=60 Nov 01 09:15:04.983247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.983185 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerDied","Data":"805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797"} Nov 01 09:15:04.983409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.983396 5710 scope.go:122] "RemoveContainer" containerID="805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797" Nov 01 09:15:04.983656 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:04.983639 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9_odh-ai-gateway-infra(dc689078-654a-4be1-8dc6-ad49900f4582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" Nov 01 09:15:04.985010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.984990 5710 scope.go:122] "RemoveContainer" containerID="8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208" Nov 01 09:15:04.985255 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:04.985234 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq_odh-ai-gateway-infra(f6d04a58-0cb7-48b0-a1ec-0d78c06601ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" Nov 01 09:15:04.986614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.986593 5710 scope.go:122] "RemoveContainer" containerID="a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d" Nov 01 09:15:04.986841 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:04.986824 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4_odh-ai-gateway-infra(5aa0cd55-b3eb-422f-87d0-f2af49632018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" Nov 01 09:15:04.993859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:04.993840 5710 scope.go:122] "RemoveContainer" containerID="8e6c76312791b31f89228ddfe0beefa85640a22221cd786f2b4ab9df2bb2cba9" Nov 01 09:15:05.736672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:05.736626 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw"] Nov 01 09:15:05.736938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:05.736885 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" containerID="cri-o://618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823" gracePeriod=30 Nov 01 09:15:06.001473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:06.001367 5710 scope.go:122] "RemoveContainer" containerID="805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797" Nov 01 09:15:06.001974 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:06.001698 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9_odh-ai-gateway-infra(dc689078-654a-4be1-8dc6-ad49900f4582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" Nov 01 09:15:06.003486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:06.003459 5710 scope.go:122] "RemoveContainer" containerID="85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d" Nov 01 09:15:06.003880 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:06.003855 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh_odh-ai-gateway-infra(9e365490-d0bf-4598-bd85-7fa0f33cf8cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" Nov 01 09:15:07.588671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:07.588633 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:15:11.006475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.006400 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:15:11.036304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.036269 5710 generic.go:356] "Generic (PLEG): container finished" podID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerID="618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823" exitCode=0 Nov 01 09:15:11.036496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.036322 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" Nov 01 09:15:11.036496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.036366 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" event={"ID":"e1dbd1a1-748a-4956-86db-8d248a00053f","Type":"ContainerDied","Data":"618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823"} Nov 01 09:15:11.036496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.036427 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw" event={"ID":"e1dbd1a1-748a-4956-86db-8d248a00053f","Type":"ContainerDied","Data":"61cd606dd442b7dfa6431c74b0c0fe2ec06e77f0a50176cb976c4c09673df77d"} Nov 01 09:15:11.036496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.036452 5710 scope.go:122] "RemoveContainer" containerID="618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823" Nov 01 09:15:11.046516 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.046497 5710 scope.go:122] "RemoveContainer" containerID="e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603" Nov 01 09:15:11.066140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.066106 5710 scope.go:122] "RemoveContainer" containerID="618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823" Nov 01 09:15:11.066562 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:11.066527 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823\": container with ID starting with 618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823 not found: ID does not exist" containerID="618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823" Nov 01 09:15:11.066701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.066568 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823"} err="failed to get container status \"618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823\": rpc error: code = NotFound desc = could not find container \"618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823\": container with ID starting with 618bba3728b7a80d3d76225cee8228eb569d1d70db462ae489ecbde39d733823 not found: ID does not exist" Nov 01 09:15:11.066701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.066590 5710 scope.go:122] "RemoveContainer" containerID="e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603" Nov 01 09:15:11.066969 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:11.066952 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603\": container with ID starting with e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603 not found: ID does not exist" containerID="e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603" Nov 01 09:15:11.067015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.066975 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603"} err="failed to get container status \"e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603\": rpc error: code = NotFound desc = could not find container \"e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603\": container with ID starting with e6dfdf773a1065450a643c962a05c1ad08ff0e96bd3b2a8eed1e413519015603 not found: ID does not exist" Nov 01 09:15:11.151800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.151759 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152004 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.151811 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152004 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.151830 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152004 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.151871 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152004 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.151931 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152045 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152087 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152085 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:15:11.152236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152186 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152223 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy\") pod \"e1dbd1a1-748a-4956-86db-8d248a00053f\" (UID: \"e1dbd1a1-748a-4956-86db-8d248a00053f\") " Nov 01 09:15:11.152236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152220 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:15:11.152541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152236 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:15:11.152541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152309 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:15:11.152639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.152617 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:11.153035 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.153008 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.153239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.153210 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.153350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.153239 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.153350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.153255 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.153350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.153270 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1dbd1a1-748a-4956-86db-8d248a00053f-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.154465 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.154438 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:15:11.154570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.154442 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:11.154570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.154498 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:15:11.154705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.154687 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr" pod "e1dbd1a1-748a-4956-86db-8d248a00053f" (UID: "e1dbd1a1-748a-4956-86db-8d248a00053f"). InnerVolumeSpecName "kube-api-access-r49vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:11.254045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.254009 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49vr\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-kube-api-access-r49vr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.254045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.254039 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.254045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.254049 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.254305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.254057 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1dbd1a1-748a-4956-86db-8d248a00053f-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:11.452392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.452348 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw"] Nov 01 09:15:11.459124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:11.459091 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9f4e9b-openshift-default-6cb8559469-kwntw"] Nov 01 09:15:13.058588 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:13.058552 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" path="/var/lib/kubelet/pods/e1dbd1a1-748a-4956-86db-8d248a00053f/volumes" Nov 01 09:15:15.479000 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:15.478961 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:15:16.051219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:16.051181 5710 scope.go:122] "RemoveContainer" containerID="8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208" Nov 01 09:15:16.051417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:16.051374 5710 scope.go:122] "RemoveContainer" containerID="805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797" Nov 01 09:15:17.055313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.055283 5710 scope.go:122] "RemoveContainer" containerID="85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d" Nov 01 09:15:17.055777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.055447 5710 scope.go:122] "RemoveContainer" containerID="a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d" Nov 01 09:15:17.079133 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.079103 5710 generic.go:356] "Generic (PLEG): container finished" podID="dc689078-654a-4be1-8dc6-ad49900f4582" containerID="5ae20a043e8f5fd4ab21ad815b58955e085b073b127e8d52e565f956c963f571" exitCode=60 Nov 01 09:15:17.079307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.079168 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerDied","Data":"5ae20a043e8f5fd4ab21ad815b58955e085b073b127e8d52e565f956c963f571"} Nov 01 09:15:17.079307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.079240 5710 scope.go:122] "RemoveContainer" containerID="805a7a1578102009abadf3d6fe6f1f9c3ad8c3434a92d0c2a573a8c7f1a85797" Nov 01 09:15:17.079711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.079687 5710 scope.go:122] "RemoveContainer" containerID="5ae20a043e8f5fd4ab21ad815b58955e085b073b127e8d52e565f956c963f571" Nov 01 09:15:17.080129 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:17.080098 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9_odh-ai-gateway-infra(dc689078-654a-4be1-8dc6-ad49900f4582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" Nov 01 09:15:17.081620 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.081597 5710 generic.go:356] "Generic (PLEG): container finished" podID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerID="b3d20662d8aca31f05b0761264a6d20ff325cbc519a98de390d311a41d049a03" exitCode=60 Nov 01 09:15:17.081711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.081663 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" event={"ID":"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba","Type":"ContainerDied","Data":"b3d20662d8aca31f05b0761264a6d20ff325cbc519a98de390d311a41d049a03"} Nov 01 09:15:17.082081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.082067 5710 scope.go:122] "RemoveContainer" containerID="b3d20662d8aca31f05b0761264a6d20ff325cbc519a98de390d311a41d049a03" Nov 01 09:15:17.082348 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:17.082330 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq_odh-ai-gateway-infra(f6d04a58-0cb7-48b0-a1ec-0d78c06601ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" Nov 01 09:15:17.107021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:17.106997 5710 scope.go:122] "RemoveContainer" containerID="8445109544f83235f685fe3f2aba720a584e6316424245d8e490046fa8c02208" Nov 01 09:15:18.091477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.091446 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerStarted","Data":"31d361b4363ed9b78e3a929120b4be0fddad2401d81b02be14c0b712019b6082"} Nov 01 09:15:18.091910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.091888 5710 scope.go:122] "RemoveContainer" containerID="31d361b4363ed9b78e3a929120b4be0fddad2401d81b02be14c0b712019b6082" Nov 01 09:15:18.092213 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:18.092187 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh_odh-ai-gateway-infra(9e365490-d0bf-4598-bd85-7fa0f33cf8cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" Nov 01 09:15:18.097099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.097076 5710 generic.go:356] "Generic (PLEG): container finished" podID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerID="3b7264226c7714065bfaab3c5a39abfc632d854b3da81b11bbfcf4607dd77134" exitCode=60 Nov 01 09:15:18.097241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.097141 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" event={"ID":"5aa0cd55-b3eb-422f-87d0-f2af49632018","Type":"ContainerDied","Data":"3b7264226c7714065bfaab3c5a39abfc632d854b3da81b11bbfcf4607dd77134"} Nov 01 09:15:18.097241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.097201 5710 scope.go:122] "RemoveContainer" containerID="a82bb7d289d545d7a1c5dd66f66cd226d26877d01003772716eca9cae9715e1d" Nov 01 09:15:18.097563 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.097547 5710 scope.go:122] "RemoveContainer" containerID="3b7264226c7714065bfaab3c5a39abfc632d854b3da81b11bbfcf4607dd77134" Nov 01 09:15:18.097868 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:18.097847 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4_odh-ai-gateway-infra(5aa0cd55-b3eb-422f-87d0-f2af49632018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" Nov 01 09:15:18.198619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.198576 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9"] Nov 01 09:15:18.306341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.306303 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq"] Nov 01 09:15:18.361099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.361059 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:18.459436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.459405 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:18.542506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.542406 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca\") pod \"dc689078-654a-4be1-8dc6-ad49900f4582\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " Nov 01 09:15:18.542669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.542607 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8\" (UniqueName: \"kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8\") pod \"dc689078-654a-4be1-8dc6-ad49900f4582\" (UID: \"dc689078-654a-4be1-8dc6-ad49900f4582\") " Nov 01 09:15:18.542824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.542797 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca" pod "dc689078-654a-4be1-8dc6-ad49900f4582" (UID: "dc689078-654a-4be1-8dc6-ad49900f4582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:18.543060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.543044 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc689078-654a-4be1-8dc6-ad49900f4582-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:18.545153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.545127 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8" pod "dc689078-654a-4be1-8dc6-ad49900f4582" (UID: "dc689078-654a-4be1-8dc6-ad49900f4582"). InnerVolumeSpecName "kube-api-access-fxhl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:18.643611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.643576 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca\") pod \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " Nov 01 09:15:18.643821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.643645 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q\" (UniqueName: \"kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q\") pod \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\" (UID: \"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba\") " Nov 01 09:15:18.644016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.643999 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxhl8\" (UniqueName: \"kubernetes.io/projected/dc689078-654a-4be1-8dc6-ad49900f4582-kube-api-access-fxhl8\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:18.644070 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.644045 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca" pod "f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" (UID: "f6d04a58-0cb7-48b0-a1ec-0d78c06601ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:18.646030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.646007 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q" pod "f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" (UID: "f6d04a58-0cb7-48b0-a1ec-0d78c06601ba"). InnerVolumeSpecName "kube-api-access-p7l7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:18.745377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.745336 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:18.745377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:18.745377 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7l7q\" (UniqueName: \"kubernetes.io/projected/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba-kube-api-access-p7l7q\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:19.106140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.106105 5710 generic.go:356] "Generic (PLEG): container finished" podID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerID="31d361b4363ed9b78e3a929120b4be0fddad2401d81b02be14c0b712019b6082" exitCode=60 Nov 01 09:15:19.106596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.106182 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerDied","Data":"31d361b4363ed9b78e3a929120b4be0fddad2401d81b02be14c0b712019b6082"} Nov 01 09:15:19.106596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.106249 5710 scope.go:122] "RemoveContainer" containerID="85431c86c48b6f2e0f8c0334ed0df8619ff83cf7641f0daf4a41c81876590d1d" Nov 01 09:15:19.106759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.106670 5710 scope.go:122] "RemoveContainer" containerID="31d361b4363ed9b78e3a929120b4be0fddad2401d81b02be14c0b712019b6082" Nov 01 09:15:19.107017 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:19.106995 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh_odh-ai-gateway-infra(9e365490-d0bf-4598-bd85-7fa0f33cf8cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" Nov 01 09:15:19.107950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.107933 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" Nov 01 09:15:19.108058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.107955 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9" event={"ID":"dc689078-654a-4be1-8dc6-ad49900f4582","Type":"ContainerDied","Data":"7989b8ebdf6c75323cd081255e92233d682ab7435d46285a44c6d552836ffe10"} Nov 01 09:15:19.109754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.109728 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" event={"ID":"f6d04a58-0cb7-48b0-a1ec-0d78c06601ba","Type":"ContainerDied","Data":"1ca030e60dcb6f4cf0e7ea290545a4ad117466c1081d30ac3ff6df7b81fecf7e"} Nov 01 09:15:19.109881 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.109771 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq" Nov 01 09:15:19.120376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.120352 5710 scope.go:122] "RemoveContainer" containerID="5ae20a043e8f5fd4ab21ad815b58955e085b073b127e8d52e565f956c963f571" Nov 01 09:15:19.130244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.130223 5710 scope.go:122] "RemoveContainer" containerID="b3d20662d8aca31f05b0761264a6d20ff325cbc519a98de390d311a41d049a03" Nov 01 09:15:19.210927 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.210892 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh"] Nov 01 09:15:19.347317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.347272 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq"] Nov 01 09:15:19.351228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.351175 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-751a73-29892075-8j9xq"] Nov 01 09:15:19.430912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.430872 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4"] Nov 01 09:15:19.455965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.455921 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9"] Nov 01 09:15:19.464667 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.464635 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892075-hkfd9"] Nov 01 09:15:19.603414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.603384 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:19.758902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.758858 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k\" (UniqueName: \"kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k\") pod \"5aa0cd55-b3eb-422f-87d0-f2af49632018\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " Nov 01 09:15:19.758902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.758907 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca\") pod \"5aa0cd55-b3eb-422f-87d0-f2af49632018\" (UID: \"5aa0cd55-b3eb-422f-87d0-f2af49632018\") " Nov 01 09:15:19.759390 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.759363 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca" pod "5aa0cd55-b3eb-422f-87d0-f2af49632018" (UID: "5aa0cd55-b3eb-422f-87d0-f2af49632018"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:19.761304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.761275 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k" pod "5aa0cd55-b3eb-422f-87d0-f2af49632018" (UID: "5aa0cd55-b3eb-422f-87d0-f2af49632018"). InnerVolumeSpecName "kube-api-access-gzn6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:19.860754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.860670 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzn6k\" (UniqueName: \"kubernetes.io/projected/5aa0cd55-b3eb-422f-87d0-f2af49632018-kube-api-access-gzn6k\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:19.860754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:19.860708 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa0cd55-b3eb-422f-87d0-f2af49632018-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:20.129879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.129777 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" Nov 01 09:15:20.129879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.129788 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4" event={"ID":"5aa0cd55-b3eb-422f-87d0-f2af49632018","Type":"ContainerDied","Data":"058d407abbcf7045a33cf213e6a33e141bd7cc2aa740db2f693aec87506a5f9f"} Nov 01 09:15:20.129879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.129871 5710 scope.go:122] "RemoveContainer" containerID="3b7264226c7714065bfaab3c5a39abfc632d854b3da81b11bbfcf4607dd77134" Nov 01 09:15:20.291426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.291397 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:20.393653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.393568 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4"] Nov 01 09:15:20.399992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.399962 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-28efa1-29892075-4hcl4"] Nov 01 09:15:20.466866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.466817 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw\" (UniqueName: \"kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw\") pod \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " Nov 01 09:15:20.467101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.466888 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca\") pod \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\" (UID: \"9e365490-d0bf-4598-bd85-7fa0f33cf8cb\") " Nov 01 09:15:20.467311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.467287 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca" pod "9e365490-d0bf-4598-bd85-7fa0f33cf8cb" (UID: "9e365490-d0bf-4598-bd85-7fa0f33cf8cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:20.467470 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.467453 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:20.469290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.469263 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw" pod "9e365490-d0bf-4598-bd85-7fa0f33cf8cb" (UID: "9e365490-d0bf-4598-bd85-7fa0f33cf8cb"). InnerVolumeSpecName "kube-api-access-z7rlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:20.568536 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:20.568501 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7rlw\" (UniqueName: \"kubernetes.io/projected/9e365490-d0bf-4598-bd85-7fa0f33cf8cb-kube-api-access-z7rlw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:21.058871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.058837 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" path="/var/lib/kubelet/pods/5aa0cd55-b3eb-422f-87d0-f2af49632018/volumes" Nov 01 09:15:21.059291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.059277 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" path="/var/lib/kubelet/pods/dc689078-654a-4be1-8dc6-ad49900f4582/volumes" Nov 01 09:15:21.059647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.059635 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" path="/var/lib/kubelet/pods/f6d04a58-0cb7-48b0-a1ec-0d78c06601ba/volumes" Nov 01 09:15:21.141128 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.141099 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" Nov 01 09:15:21.141605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.141134 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh" event={"ID":"9e365490-d0bf-4598-bd85-7fa0f33cf8cb","Type":"ContainerDied","Data":"a8e08a442f62c7d31f6e3c3b02f65cd53fdd54e635558d51916ac91cace69a5e"} Nov 01 09:15:21.141605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.141196 5710 scope.go:122] "RemoveContainer" containerID="31d361b4363ed9b78e3a929120b4be0fddad2401d81b02be14c0b712019b6082" Nov 01 09:15:21.294712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.294676 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh"] Nov 01 09:15:21.303225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:21.303184 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6752e4-29892075-jcckh"] Nov 01 09:15:23.058089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.058054 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" path="/var/lib/kubelet/pods/9e365490-d0bf-4598-bd85-7fa0f33cf8cb/volumes" Nov 01 09:15:23.522252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522215 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp"] Nov 01 09:15:23.522781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522765 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522783 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522795 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522800 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522809 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522814 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522828 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.522834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522834 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522843 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522848 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522856 5710 cpu_manager.go:414] "containerMap: removing container" podUID="af4645f8-6919-4682-ad2d-738ad3dd6dae" containerName="payload-pre-processing" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522861 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4645f8-6919-4682-ad2d-738ad3dd6dae" containerName="payload-pre-processing" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522878 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522883 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522890 5710 cpu_manager.go:414] "containerMap: removing container" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522895 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522901 5710 cpu_manager.go:414] "containerMap: removing container" podUID="691b3adb-9c90-4da5-9327-0351757a7dcd" containerName="payload-processing" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522906 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691b3adb-9c90-4da5-9327-0351757a7dcd" containerName="payload-processing" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522914 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522919 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522927 5710 cpu_manager.go:414] "containerMap: removing container" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522932 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522937 5710 cpu_manager.go:414] "containerMap: removing container" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522942 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522951 5710 cpu_manager.go:414] "containerMap: removing container" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522956 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522964 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.522968 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523037 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523046 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523051 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523060 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:23.523053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523064 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523069 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4645f8-6919-4682-ad2d-738ad3dd6dae" containerName="payload-pre-processing" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523077 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523083 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523088 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="691b3adb-9c90-4da5-9327-0351757a7dcd" containerName="payload-processing" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523094 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523098 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.524071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.523103 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa0cd55-b3eb-422f-87d0-f2af49632018" containerName="cleanup" Nov 01 09:15:23.527733 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.527694 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.578615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.578573 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c1c173-serving-cert\"" Nov 01 09:15:23.604464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.604414 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.604688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.604518 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5qf\" (UniqueName: \"kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.604688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.604594 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.604688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.604637 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.627434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.627396 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp"] Nov 01 09:15:23.705434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.705387 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5qf\" (UniqueName: \"kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.705645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.705461 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.705645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.705498 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.705783 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.705655 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.705783 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:23.705661 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-serving-cert: secret "maas-api-e2e-praxis-b-c1c173-serving-cert" not found Nov 01 09:15:23.705783 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:23.705759 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls podName:137d3e78-18a3-4e77-93a2-33baa71eabaa nodeName:}" failed. No retries permitted until 2026-11-01 09:15:24.205708587 +0000 UTC m=+2547.294045268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls") pod "maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" (UID: "137d3e78-18a3-4e77-93a2-33baa71eabaa") : secret "maas-api-e2e-praxis-b-c1c173-serving-cert" not found Nov 01 09:15:23.706356 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.706336 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.708306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.708276 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:23.784320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:23.778115 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5qf\" (UniqueName: \"kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:24.211266 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:24.211219 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:24.214079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:24.214053 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:24.442938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:24.442900 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:24.828651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:24.828567 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp"] Nov 01 09:15:24.836382 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:24.836313 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137d3e78_18a3_4e77_93a2_33baa71eabaa.slice/crio-06212cb1899c9053083126eba61f63108b357be8edfea13a5028b98c3c5b37f3 WatchSource:0}: Error finding container 06212cb1899c9053083126eba61f63108b357be8edfea13a5028b98c3c5b37f3: Status 404 returned error can't find the container with id 06212cb1899c9053083126eba61f63108b357be8edfea13a5028b98c3c5b37f3 Nov 01 09:15:25.168334 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:25.168296 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" event={"ID":"137d3e78-18a3-4e77-93a2-33baa71eabaa","Type":"ContainerStarted","Data":"06212cb1899c9053083126eba61f63108b357be8edfea13a5028b98c3c5b37f3"} Nov 01 09:15:26.175284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:26.175248 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" event={"ID":"137d3e78-18a3-4e77-93a2-33baa71eabaa","Type":"ContainerStarted","Data":"ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4"} Nov 01 09:15:26.175703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:26.175310 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:26.287561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:26.287503 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" podStartSLOduration=2.355078864 podStartE2EDuration="3.287485795s" totalImagesPullingTime="932.406931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:26.285444394 +0000 UTC m=+2549.373781079" watchObservedRunningTime="2026-11-01 09:15:26.287485795 +0000 UTC m=+2549.375822467" Nov 01 09:15:32.187277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:32.187242 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:15:34.357344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:34.357313 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:15:38.123634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.123589 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8"] Nov 01 09:15:38.124385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124361 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" Nov 01 09:15:38.124484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124387 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" Nov 01 09:15:38.124484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124437 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:38.124484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124445 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:38.124757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124565 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d04a58-0cb7-48b0-a1ec-0d78c06601ba" containerName="cleanup" Nov 01 09:15:38.124757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124578 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1dbd1a1-748a-4956-86db-8d248a00053f" containerName="istio-proxy" Nov 01 09:15:38.124757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124593 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e365490-d0bf-4598-bd85-7fa0f33cf8cb" containerName="cleanup" Nov 01 09:15:38.124757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.124605 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc689078-654a-4be1-8dc6-ad49900f4582" containerName="cleanup" Nov 01 09:15:38.128401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.128376 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.226623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.226582 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-514eb3-serving-cert\"" Nov 01 09:15:38.229033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.229009 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8"] Nov 01 09:15:38.267245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.267198 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.267421 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.267372 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxqg\" (UniqueName: \"kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.267473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.267455 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.267518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.267492 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.369033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.368976 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxqg\" (UniqueName: \"kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.369260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.369134 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.369260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.369170 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.369381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.369298 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.372414 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:38.370282 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-serving-cert: secret "maas-api-e2e-praxis-b-514eb3-serving-cert" not found Nov 01 09:15:38.372414 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:38.370394 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls podName:44712043-211d-44d7-81a6-00e4b52b24e0 nodeName:}" failed. No retries permitted until 2026-11-01 09:15:38.870368745 +0000 UTC m=+2561.958705399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls") pod "maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" (UID: "44712043-211d-44d7-81a6-00e4b52b24e0") : secret "maas-api-e2e-praxis-b-514eb3-serving-cert" not found Nov 01 09:15:38.372414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.372002 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.376419 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.376291 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.431284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.431253 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxqg\" (UniqueName: \"kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.875641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.875597 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:38.878365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:38.878332 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:39.041886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:39.041848 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:39.444527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:39.444435 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8"] Nov 01 09:15:39.451200 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:39.451160 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44712043_211d_44d7_81a6_00e4b52b24e0.slice/crio-f12a3b3584d649dd7559889615de6700567fdce7336e226236d8dbbed45d2eeb WatchSource:0}: Error finding container f12a3b3584d649dd7559889615de6700567fdce7336e226236d8dbbed45d2eeb: Status 404 returned error can't find the container with id f12a3b3584d649dd7559889615de6700567fdce7336e226236d8dbbed45d2eeb Nov 01 09:15:40.262682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:40.262645 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" event={"ID":"44712043-211d-44d7-81a6-00e4b52b24e0","Type":"ContainerStarted","Data":"f12a3b3584d649dd7559889615de6700567fdce7336e226236d8dbbed45d2eeb"} Nov 01 09:15:41.271439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:41.271401 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" event={"ID":"44712043-211d-44d7-81a6-00e4b52b24e0","Type":"ContainerStarted","Data":"66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688"} Nov 01 09:15:41.271907 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:41.271480 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:41.382832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:41.382776 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" podStartSLOduration=2.481372485 podStartE2EDuration="3.382763226s" totalImagesPullingTime="901.390741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:41.381382916 +0000 UTC m=+2564.469719600" watchObservedRunningTime="2026-11-01 09:15:41.382763226 +0000 UTC m=+2564.471099919" Nov 01 09:15:43.017937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.017884 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf"] Nov 01 09:15:43.018433 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.018127 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" podUID="9da49772-4bf4-42eb-9a19-8b91024cc1e2" containerName="payload-processing" containerID="cri-o://122037902d7b4e0acdf707788514c36a291694dda56f53df833e28bbdccb69c9" gracePeriod=30 Nov 01 09:15:43.285064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.285034 5710 generic.go:356] "Generic (PLEG): container finished" podID="9da49772-4bf4-42eb-9a19-8b91024cc1e2" containerID="122037902d7b4e0acdf707788514c36a291694dda56f53df833e28bbdccb69c9" exitCode=0 Nov 01 09:15:43.285247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.285111 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" event={"ID":"9da49772-4bf4-42eb-9a19-8b91024cc1e2","Type":"ContainerDied","Data":"122037902d7b4e0acdf707788514c36a291694dda56f53df833e28bbdccb69c9"} Nov 01 09:15:43.285247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.285182 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" event={"ID":"9da49772-4bf4-42eb-9a19-8b91024cc1e2","Type":"ContainerDied","Data":"e5f71a9fcb96fac1cc9322062955c4fe53b93f24921b94de49d3e2dc2dfa071d"} Nov 01 09:15:43.285247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.285206 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f71a9fcb96fac1cc9322062955c4fe53b93f24921b94de49d3e2dc2dfa071d" Nov 01 09:15:43.291758 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.291734 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:15:43.430020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.429981 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume\") pod \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " Nov 01 09:15:43.430251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.430043 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl\" (UniqueName: \"kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl\") pod \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\" (UID: \"9da49772-4bf4-42eb-9a19-8b91024cc1e2\") " Nov 01 09:15:43.430446 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.430416 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume" pod "9da49772-4bf4-42eb-9a19-8b91024cc1e2" (UID: "9da49772-4bf4-42eb-9a19-8b91024cc1e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:15:43.430653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.430632 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9da49772-4bf4-42eb-9a19-8b91024cc1e2-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:43.433883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.433852 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl" pod "9da49772-4bf4-42eb-9a19-8b91024cc1e2" (UID: "9da49772-4bf4-42eb-9a19-8b91024cc1e2"). InnerVolumeSpecName "kube-api-access-dhcpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:15:43.535824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:43.532095 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcpl\" (UniqueName: \"kubernetes.io/projected/9da49772-4bf4-42eb-9a19-8b91024cc1e2-kube-api-access-dhcpl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:15:44.292014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:44.291969 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf" Nov 01 09:15:44.448367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:44.448299 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf"] Nov 01 09:15:44.492261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:44.492216 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be7cc-5889688f8d-zvbjf"] Nov 01 09:15:45.058384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:45.058349 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da49772-4bf4-42eb-9a19-8b91024cc1e2" path="/var/lib/kubelet/pods/9da49772-4bf4-42eb-9a19-8b91024cc1e2/volumes" Nov 01 09:15:45.478441 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:15:45.478400 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:15:47.283433 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:47.283404 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:15:49.172188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.172137 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq"] Nov 01 09:15:49.172757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.172733 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9da49772-4bf4-42eb-9a19-8b91024cc1e2" containerName="payload-processing" Nov 01 09:15:49.172757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.172752 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da49772-4bf4-42eb-9a19-8b91024cc1e2" containerName="payload-processing" Nov 01 09:15:49.172894 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.172884 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da49772-4bf4-42eb-9a19-8b91024cc1e2" containerName="payload-processing" Nov 01 09:15:49.177477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.177455 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.227923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.227890 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0468a5-openshift-default-dockercfg-xwmlz\"" Nov 01 09:15:49.260741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.256018 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq"] Nov 01 09:15:49.303258 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.303199 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.303601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.303517 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.303868 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.303907 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.303971 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.304029 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.304099 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.304249 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.304428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.304279 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406082 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406039 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406117 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406143 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406197 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406247 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406280 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406414 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406441 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406482 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406698 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406840 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406817 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.406964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.406875 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.407122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.407099 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.407460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.407436 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.408679 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.408653 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.408989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.408966 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.464120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.464032 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.464304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.464237 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd\") pod \"e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.491763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.491699 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:49.950294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.950252 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq"] Nov 01 09:15:49.957538 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:49.957508 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930bec2c_702a_4b9f_aa04_c1afb3072c24.slice/crio-04a50653dcf53861a6b637fc951abd32c4238bd4a02aedf12ad5e5c9cf00df53 WatchSource:0}: Error finding container 04a50653dcf53861a6b637fc951abd32c4238bd4a02aedf12ad5e5c9cf00df53: Status 404 returned error can't find the container with id 04a50653dcf53861a6b637fc951abd32c4238bd4a02aedf12ad5e5c9cf00df53 Nov 01 09:15:49.960077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.960039 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:15:49.960385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.960115 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:15:49.960385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:49.960151 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:15:50.356307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:50.356270 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" event={"ID":"930bec2c-702a-4b9f-aa04-c1afb3072c24","Type":"ContainerStarted","Data":"3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f"} Nov 01 09:15:50.356793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:50.356337 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" event={"ID":"930bec2c-702a-4b9f-aa04-c1afb3072c24","Type":"ContainerStarted","Data":"04a50653dcf53861a6b637fc951abd32c4238bd4a02aedf12ad5e5c9cf00df53"} Nov 01 09:15:50.471982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:50.471917 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podStartSLOduration=1.471899696 podStartE2EDuration="1.471899696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:50.470079426 +0000 UTC m=+2573.558416135" watchObservedRunningTime="2026-11-01 09:15:50.471899696 +0000 UTC m=+2573.560236362" Nov 01 09:15:50.491834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:50.491796 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:51.499339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.499302 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:51.604168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.604125 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74"] Nov 01 09:15:51.608498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.608468 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.658474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.658435 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-54e01b-openshift-default-dockercfg-gln2n\"" Nov 01 09:15:51.711587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.711553 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74"] Nov 01 09:15:51.732099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732067 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732104 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732209 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732327 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732532 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732414 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732532 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732471 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732532 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732522 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732608 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhws\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.732702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.732672 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.833825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.833699 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhws\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.833825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.833783 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.833825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.833815 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.833836 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.833972 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834090 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834158 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834208 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834280 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834287 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834346 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.834668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.834531 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.835058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.835015 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.835058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.835049 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.836647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.836622 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.837069 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.837052 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.891533 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.891497 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.891822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.891791 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhws\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:51.924263 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:51.924227 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:52.360457 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:52.360418 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74"] Nov 01 09:15:52.367866 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:15:52.367826 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bea5470_78e2_4de4_966a_548c85083527.slice/crio-22163fc254a296684862c53d4ef7f557e40f310076b1326adb91e8fd02281112 WatchSource:0}: Error finding container 22163fc254a296684862c53d4ef7f557e40f310076b1326adb91e8fd02281112: Status 404 returned error can't find the container with id 22163fc254a296684862c53d4ef7f557e40f310076b1326adb91e8fd02281112 Nov 01 09:15:52.368988 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:52.368961 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:52.370122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:52.370103 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:15:52.370418 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:52.370390 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:15:52.370494 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:52.370473 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:15:52.370562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:52.370516 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:15:53.375632 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:53.375594 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" event={"ID":"6bea5470-78e2-4de4-966a-548c85083527","Type":"ContainerStarted","Data":"f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc"} Nov 01 09:15:53.376054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:53.375646 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" event={"ID":"6bea5470-78e2-4de4-966a-548c85083527","Type":"ContainerStarted","Data":"22163fc254a296684862c53d4ef7f557e40f310076b1326adb91e8fd02281112"} Nov 01 09:15:53.536379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:53.536293 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podStartSLOduration=2.536272195 podStartE2EDuration="2.536272195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:15:53.534754096 +0000 UTC m=+2576.623090763" watchObservedRunningTime="2026-11-01 09:15:53.536272195 +0000 UTC m=+2576.624608874" Nov 01 09:15:53.925115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:53.925065 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:15:53.926745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:53.926695 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 09:15:53.926898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:53.926787 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 09:15:54.924685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:54.924648 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 09:15:54.925215 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:54.924748 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 09:15:55.925585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:55.925544 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 09:15:55.926155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:55.925624 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 09:15:56.934774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:56.925237 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 09:15:56.934774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:56.925311 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 09:15:57.250482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:57.250407 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd"] Nov 01 09:15:57.250641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:57.250621 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" containerID="cri-o://4ad6a820ceb82c1f4248bee29973cc44475dfb51b55d8801876c78b8ecd7f175" gracePeriod=30 Nov 01 09:15:57.342246 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:57.342199 5710 patch_prober.go:29] interesting pod/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:15:57.344770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:57.342295 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:15:57.925397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:57.925345 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 09:15:57.925609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:57.925414 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 09:15:58.924842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:58.924796 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 09:15:58.925254 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:15:58.924881 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 09:16:00.280279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:00.280183 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:16:00.280874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:00.280844 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:16:00.281974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:00.281950 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:16:02.464144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.464112 5710 generic.go:356] "Generic (PLEG): container finished" podID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerID="4ad6a820ceb82c1f4248bee29973cc44475dfb51b55d8801876c78b8ecd7f175" exitCode=0 Nov 01 09:16:02.464677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.464186 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" event={"ID":"d2e16e5c-c01c-4246-a0f2-7f1179e5951c","Type":"ContainerDied","Data":"4ad6a820ceb82c1f4248bee29973cc44475dfb51b55d8801876c78b8ecd7f175"} Nov 01 09:16:02.464677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.464258 5710 scope.go:122] "RemoveContainer" containerID="b5ea5fbbdf83750a660d098b74d5fca04ce170de1858dc44302c34cdf043ea75" Nov 01 09:16:02.592221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.592190 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:16:02.629608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.629573 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.629843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.629615 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.629843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.629685 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.629843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.629710 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.629843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.629836 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.630071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.629978 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.630071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630005 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:16:02.630071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630029 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.630227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630141 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.630227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630181 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs\") pod \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\" (UID: \"d2e16e5c-c01c-4246-a0f2-7f1179e5951c\") " Nov 01 09:16:02.630340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630289 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:16:02.630510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630479 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:16:02.630818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630522 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:16:02.630934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630779 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:16:02.631019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.630891 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.631134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.631028 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.632802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.632710 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:16:02.633064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.633034 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "kube-api-access-ql2fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:16:02.633169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.633058 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:16:02.633169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.633129 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token" pod "d2e16e5c-c01c-4246-a0f2-7f1179e5951c" (UID: "d2e16e5c-c01c-4246-a0f2-7f1179e5951c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:16:02.732691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732656 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql2fn\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-kube-api-access-ql2fn\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.732691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732686 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.732691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732696 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.733015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732705 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.733015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732744 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.733015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732753 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:02.733015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:02.732762 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e16e5c-c01c-4246-a0f2-7f1179e5951c-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:03.472630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:03.472599 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" Nov 01 09:16:03.472630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:03.472616 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd" event={"ID":"d2e16e5c-c01c-4246-a0f2-7f1179e5951c","Type":"ContainerDied","Data":"e37ee007774ec4d2403bee4ac41ec157f4d5b8b0de9119aa5f24cc4d427fc647"} Nov 01 09:16:03.473225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:03.472692 5710 scope.go:122] "RemoveContainer" containerID="4ad6a820ceb82c1f4248bee29973cc44475dfb51b55d8801876c78b8ecd7f175" Nov 01 09:16:03.629538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:03.629486 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd"] Nov 01 09:16:03.674355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:03.674310 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-55644e-openshift-default-585ff9bf69-z2zpd"] Nov 01 09:16:05.058616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:05.058574 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" path="/var/lib/kubelet/pods/d2e16e5c-c01c-4246-a0f2-7f1179e5951c/volumes" Nov 01 09:16:15.479450 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:16:15.479408 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:16:28.764756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:28.764702 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:16:40.925631 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.925542 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r"] Nov 01 09:16:40.926189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.926167 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" Nov 01 09:16:40.926255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.926192 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" Nov 01 09:16:40.926255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.926212 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" Nov 01 09:16:40.926255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.926221 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" Nov 01 09:16:40.926488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.926349 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" Nov 01 09:16:40.926488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.926366 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e16e5c-c01c-4246-a0f2-7f1179e5951c" containerName="istio-proxy" Nov 01 09:16:40.931097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.931076 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:40.984097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.984057 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r"] Nov 01 09:16:40.987669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.987632 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:40.987873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.987815 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:40.987965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.987940 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:40.988098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.988080 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:40.988152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.988138 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:40.988245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:40.988222 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfqlj\" (UniqueName: \"kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.027485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.027447 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-6752e4\"/\"kube-root-ca.crt\"" Nov 01 09:16:41.027681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.027550 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-6752e4\"/\"praxis-route-6752e4-kserve-self-signed-certs\"" Nov 01 09:16:41.027681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.027577 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-6752e4\"/\"default-dockercfg-k4thl\"" Nov 01 09:16:41.027681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.027577 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-6752e4\"/\"openshift-service-ca.crt\"" Nov 01 09:16:41.089239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089203 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089257 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089313 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089591 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089448 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089591 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089482 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfqlj\" (UniqueName: \"kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089591 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089577 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089752 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.089848 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089807 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.090010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.089982 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.092018 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.091999 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.092333 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.092311 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.147979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.147936 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfqlj\" (UniqueName: \"kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj\") pod \"praxis-route-6752e4-kserve-7db4b6c98c-shq7r\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.244858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.244770 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:41.634052 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.634011 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r"] Nov 01 09:16:41.715672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:41.715635 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" event={"ID":"b601ac90-0ffc-4bdd-9b32-9f10356af28d","Type":"ContainerStarted","Data":"45c319dabc945eb6f20a9fa989f9d596226dfaa41dc4d61600683c02ecfa323f"} Nov 01 09:16:42.723793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:42.723752 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" event={"ID":"b601ac90-0ffc-4bdd-9b32-9f10356af28d","Type":"ContainerStarted","Data":"d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb"} Nov 01 09:16:42.834403 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:42.834340 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" podStartSLOduration=2.8343249569999998 podStartE2EDuration="2.834324957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:42.832893621 +0000 UTC m=+2625.921230307" watchObservedRunningTime="2026-11-01 09:16:42.834324957 +0000 UTC m=+2625.922661628" Nov 01 09:16:43.245916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:43.245876 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:44.063857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:44.063820 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2"] Nov 01 09:16:45.477682 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:16:45.477637 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:16:46.067582 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.067533 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68"] Nov 01 09:16:46.072008 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.071983 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.147859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.147817 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpddn\" (UniqueName: \"kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.148081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.147889 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.148159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.148087 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.148248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.148219 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.169493 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.169450 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0468a5-serving-cert\"" Nov 01 09:16:46.171697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.171672 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68"] Nov 01 09:16:46.249316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.249274 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpddn\" (UniqueName: \"kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.249552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.249348 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.249552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.249451 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.249552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.249500 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.249787 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:16:46.249593 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-serving-cert: secret "maas-api-e2e-praxis-a-0468a5-serving-cert" not found Nov 01 09:16:46.249787 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:16:46.249660 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls podName:25b87df0-e39e-40bc-9500-09807efa8334 nodeName:}" failed. No retries permitted until 2026-11-01 09:16:46.74964318 +0000 UTC m=+2629.837979830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls") pod "maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" (UID: "25b87df0-e39e-40bc-9500-09807efa8334") : secret "maas-api-e2e-praxis-a-0468a5-serving-cert" not found Nov 01 09:16:46.250405 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.250376 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.252161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.252136 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.308125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.308087 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpddn\" (UniqueName: \"kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.760649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.760602 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.763453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.763431 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0468a5-8574958446-x9d68\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:46.988647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:46.988607 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:47.367912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:47.367871 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68"] Nov 01 09:16:47.375634 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:16:47.375591 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b87df0_e39e_40bc_9500_09807efa8334.slice/crio-98534d3e9dc12018cfbd78e845ab3466ce9dbba169a16c273ec279423ac12b72 WatchSource:0}: Error finding container 98534d3e9dc12018cfbd78e845ab3466ce9dbba169a16c273ec279423ac12b72: Status 404 returned error can't find the container with id 98534d3e9dc12018cfbd78e845ab3466ce9dbba169a16c273ec279423ac12b72 Nov 01 09:16:47.377186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:47.377167 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:16:47.755187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:47.755085 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" event={"ID":"25b87df0-e39e-40bc-9500-09807efa8334","Type":"ContainerStarted","Data":"98534d3e9dc12018cfbd78e845ab3466ce9dbba169a16c273ec279423ac12b72"} Nov 01 09:16:48.766030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:48.765941 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" event={"ID":"25b87df0-e39e-40bc-9500-09807efa8334","Type":"ContainerStarted","Data":"f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419"} Nov 01 09:16:48.766419 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:48.766050 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:48.878063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:48.878007 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" podStartSLOduration=1.8675562079999999 podStartE2EDuration="2.877990782s" totalImagesPullingTime="1.010434574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:48.877632919 +0000 UTC m=+2631.965969591" watchObservedRunningTime="2026-11-01 09:16:48.877990782 +0000 UTC m=+2631.966327455" Nov 01 09:16:49.065221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.065122 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" podUID="036a5542-40c4-4f8a-9dbf-bd611c41f0f5" containerName="payload-processing" containerID="cri-o://d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53" gracePeriod=25 Nov 01 09:16:49.359578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.359548 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:16:49.509704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.509669 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5\" (UniqueName: \"kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5\") pod \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " Nov 01 09:16:49.509929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.509818 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume\") pod \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\" (UID: \"036a5542-40c4-4f8a-9dbf-bd611c41f0f5\") " Nov 01 09:16:49.510234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.510208 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume" pod "036a5542-40c4-4f8a-9dbf-bd611c41f0f5" (UID: "036a5542-40c4-4f8a-9dbf-bd611c41f0f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:16:49.512281 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.512258 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5" pod "036a5542-40c4-4f8a-9dbf-bd611c41f0f5" (UID: "036a5542-40c4-4f8a-9dbf-bd611c41f0f5"). InnerVolumeSpecName "kube-api-access-flkw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:16:49.611414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.611376 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:49.611414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.611409 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flkw5\" (UniqueName: \"kubernetes.io/projected/036a5542-40c4-4f8a-9dbf-bd611c41f0f5-kube-api-access-flkw5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:16:49.773867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.773828 5710 generic.go:356] "Generic (PLEG): container finished" podID="036a5542-40c4-4f8a-9dbf-bd611c41f0f5" containerID="d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53" exitCode=0 Nov 01 09:16:49.774387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.773890 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" Nov 01 09:16:49.774387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.773925 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" event={"ID":"036a5542-40c4-4f8a-9dbf-bd611c41f0f5","Type":"ContainerDied","Data":"d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53"} Nov 01 09:16:49.774387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.773990 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2" event={"ID":"036a5542-40c4-4f8a-9dbf-bd611c41f0f5","Type":"ContainerDied","Data":"3689a2f33e74be6d0dd965b00d9f17f38102d2d15228fd0b10c697e4ad3e7366"} Nov 01 09:16:49.774387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.774021 5710 scope.go:122] "RemoveContainer" containerID="d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53" Nov 01 09:16:49.784178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.784155 5710 scope.go:122] "RemoveContainer" containerID="d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53" Nov 01 09:16:49.784465 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:16:49.784446 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53\": container with ID starting with d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53 not found: ID does not exist" containerID="d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53" Nov 01 09:16:49.784530 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.784477 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53"} err="failed to get container status \"d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53\": rpc error: code = NotFound desc = could not find container \"d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53\": container with ID starting with d48f9a25740cd5836271804825dc44dfe10f84695a998990b1cf3225ffee5b53 not found: ID does not exist" Nov 01 09:16:49.927607 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.927526 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2"] Nov 01 09:16:49.931567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:49.931530 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-28efa1-7b57675d77-wcph2"] Nov 01 09:16:51.057874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:51.057842 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036a5542-40c4-4f8a-9dbf-bd611c41f0f5" path="/var/lib/kubelet/pods/036a5542-40c4-4f8a-9dbf-bd611c41f0f5/volumes" Nov 01 09:16:51.245328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:51.245287 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:51.257895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:51.257867 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:51.804981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:51.804946 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:16:52.055740 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.055643 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb"] Nov 01 09:16:52.056197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.056183 5710 cpu_manager.go:414] "containerMap: removing container" podUID="036a5542-40c4-4f8a-9dbf-bd611c41f0f5" containerName="payload-processing" Nov 01 09:16:52.056245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.056198 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036a5542-40c4-4f8a-9dbf-bd611c41f0f5" containerName="payload-processing" Nov 01 09:16:52.056306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.056298 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="036a5542-40c4-4f8a-9dbf-bd611c41f0f5" containerName="payload-processing" Nov 01 09:16:52.059786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.059759 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.108678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.108639 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b711ae-openshift-default-dockercfg-ms66n\"" Nov 01 09:16:52.116800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.116767 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb"] Nov 01 09:16:52.145447 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145390 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.145671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145458 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.145671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145559 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.145671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145611 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrvk\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.145897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145789 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.145897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145834 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.146024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145918 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.146024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.145939 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.146143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.146035 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247249 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247297 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247329 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrvk\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247397 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247430 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247465 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247492 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247552 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.247939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.247657 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.248099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.248074 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.248179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.248047 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.248216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.248181 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.248286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.248263 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.248286 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.248277 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.250545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.250522 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.250777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.250761 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.333794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.320140 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.338257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.338182 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrvk\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.377792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.377666 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:52.834052 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.834009 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb"] Nov 01 09:16:52.841308 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:16:52.841271 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea95f58f_1bc9_41c2_885a_d7f90b862454.slice/crio-8f9022c6758ccc27f2a7ed18e8071a1f31125bb05afbe3f2d167a8b7c1943d3c WatchSource:0}: Error finding container 8f9022c6758ccc27f2a7ed18e8071a1f31125bb05afbe3f2d167a8b7c1943d3c: Status 404 returned error can't find the container with id 8f9022c6758ccc27f2a7ed18e8071a1f31125bb05afbe3f2d167a8b7c1943d3c Nov 01 09:16:52.844302 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.844266 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:16:52.844417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.844339 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:16:52.844417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:52.844368 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:16:53.820661 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:53.820615 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" event={"ID":"ea95f58f-1bc9-41c2-885a-d7f90b862454","Type":"ContainerStarted","Data":"3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed"} Nov 01 09:16:53.821444 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:53.821412 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" event={"ID":"ea95f58f-1bc9-41c2-885a-d7f90b862454","Type":"ContainerStarted","Data":"8f9022c6758ccc27f2a7ed18e8071a1f31125bb05afbe3f2d167a8b7c1943d3c"} Nov 01 09:16:54.037861 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:54.037789 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podStartSLOduration=2.037765463 podStartE2EDuration="2.037765463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:16:53.990272664 +0000 UTC m=+2637.078609336" watchObservedRunningTime="2026-11-01 09:16:54.037765463 +0000 UTC m=+2637.126102136" Nov 01 09:16:54.386066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:54.386027 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:16:54.391239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:54.390870 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 09:16:54.391239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:54.390941 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 09:16:54.779953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:54.779873 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:16:55.378264 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:55.378229 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 09:16:55.378660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:55.378283 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 09:16:56.379145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:56.379106 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 09:16:56.379600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:56.379177 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 09:16:57.379168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:57.379125 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 09:16:57.379168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:57.379201 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 09:16:58.378624 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:58.378581 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 09:16:58.378824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:58.378645 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 09:16:59.379149 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:59.379107 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 09:16:59.379593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:16:59.379202 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 09:17:00.062083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:00.062040 5710 scope.go:122] "RemoveContainer" containerID="122037902d7b4e0acdf707788514c36a291694dda56f53df833e28bbdccb69c9" Nov 01 09:17:00.879675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:00.879639 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:17:00.880213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:00.880042 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:17:00.881051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:00.881031 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:17:06.699301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:06.699260 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr"] Nov 01 09:17:06.699928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:06.699563 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" containerID="cri-o://80df6e6c7c6a1637ed19546ff4e5aabf2d9f7154b83ff48dfd6ce302141c2aa9" gracePeriod=30 Nov 01 09:17:07.193961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.193918 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks"] Nov 01 09:17:07.200132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.200086 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.268492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.268458 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.268776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.268505 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.268776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.268537 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.268776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.268656 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.268776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.268710 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.269016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.268880 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.269064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.269027 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.269126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.269069 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.269203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.269186 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46b6\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.279962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.279928 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks"] Nov 01 09:17:07.370627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370585 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.370849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370634 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.370849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370690 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.370849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370753 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.370849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370842 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370875 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370952 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46b6\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.371088 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.371154 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.370992 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.371234 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.371275 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.371539 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.371628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.371579 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.373628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.373606 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.373954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.373937 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.428958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.428917 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.429132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.429011 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46b6\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.515940 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.515838 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:07.974476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.974429 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks"] Nov 01 09:17:07.981848 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:17:07.981812 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18480453_3419_4ee8_8d6e_6af8aa27537e.slice/crio-e1836339632fb234a827bbe383924ce1d16c6af9dd40d1d5ac55f3c59c09d586 WatchSource:0}: Error finding container e1836339632fb234a827bbe383924ce1d16c6af9dd40d1d5ac55f3c59c09d586: Status 404 returned error can't find the container with id e1836339632fb234a827bbe383924ce1d16c6af9dd40d1d5ac55f3c59c09d586 Nov 01 09:17:07.984560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.984523 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:07.984698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.984636 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:07.984698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:07.984682 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:08.929281 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:08.929236 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" event={"ID":"18480453-3419-4ee8-8d6e-6af8aa27537e","Type":"ContainerStarted","Data":"9c16c98ea5e1f81db6c089041cb6aff6985738e906813e72d71e29f413e09cb4"} Nov 01 09:17:08.929523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:08.929345 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" event={"ID":"18480453-3419-4ee8-8d6e-6af8aa27537e","Type":"ContainerStarted","Data":"e1836339632fb234a827bbe383924ce1d16c6af9dd40d1d5ac55f3c59c09d586"} Nov 01 09:17:09.126808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:09.126744 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podStartSLOduration=2.126727333 podStartE2EDuration="2.126727333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:09.106345092 +0000 UTC m=+2652.194681763" watchObservedRunningTime="2026-11-01 09:17:09.126727333 +0000 UTC m=+2652.215063996" Nov 01 09:17:09.516124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:09.516085 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:09.517993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:09.517967 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 09:17:09.518080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:09.518039 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 09:17:10.516220 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:10.516183 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 09:17:10.516667 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:10.516254 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 09:17:10.806876 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:10.806767 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:17:10.806876 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:10.806840 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:17:11.516914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:11.516877 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 09:17:11.517356 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:11.516932 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 09:17:11.956048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:11.956016 5710 generic.go:356] "Generic (PLEG): container finished" podID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerID="80df6e6c7c6a1637ed19546ff4e5aabf2d9f7154b83ff48dfd6ce302141c2aa9" exitCode=0 Nov 01 09:17:11.956236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:11.956086 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" event={"ID":"7fe63c5e-6364-43ed-9d85-781e4f30ab3a","Type":"ContainerDied","Data":"80df6e6c7c6a1637ed19546ff4e5aabf2d9f7154b83ff48dfd6ce302141c2aa9"} Nov 01 09:17:11.956236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:11.956155 5710 scope.go:122] "RemoveContainer" containerID="9b4fcd38b2a866e5731c5a95cb70b4246c5b90c2e503df54e8a95e090597a183" Nov 01 09:17:12.022931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.022905 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:17:12.146311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146274 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146321 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146383 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146439 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146610 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:17:12.146756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146624 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146699 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:17:12.146756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146702 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146815 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:17:12.146995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146823 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146882 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:17:12.146995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146902 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.146995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146906 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:17:12.146995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.146939 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token\") pod \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\" (UID: \"7fe63c5e-6364-43ed-9d85-781e4f30ab3a\") " Nov 01 09:17:12.147402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.147382 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.147636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.147405 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.147636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.147421 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.147636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.147436 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.147636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.147444 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.149028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.149002 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "kube-api-access-8cq2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:17:12.149179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.149159 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:17:12.149259 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.149208 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:17:12.149259 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.149218 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy" pod "7fe63c5e-6364-43ed-9d85-781e4f30ab3a" (UID: "7fe63c5e-6364-43ed-9d85-781e4f30ab3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:17:12.249160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.249027 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cq2t\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-kube-api-access-8cq2t\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.249160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.249066 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.249160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.249076 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.249160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.249085 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe63c5e-6364-43ed-9d85-781e4f30ab3a-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:12.516472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.516373 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 09:17:12.516472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.516447 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 09:17:12.964028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.963998 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" Nov 01 09:17:12.964511 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.963995 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr" event={"ID":"7fe63c5e-6364-43ed-9d85-781e4f30ab3a","Type":"ContainerDied","Data":"804071d02176e1b07c27e4fc7744bf9ad0c70441556475a7f7fca0e9c2b97d83"} Nov 01 09:17:12.964511 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:12.964145 5710 scope.go:122] "RemoveContainer" containerID="80df6e6c7c6a1637ed19546ff4e5aabf2d9f7154b83ff48dfd6ce302141c2aa9" Nov 01 09:17:13.170455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:13.169184 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr"] Nov 01 09:17:13.211568 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:13.211536 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-28efa1-openshift-default-d898cb8d5-6crgr"] Nov 01 09:17:13.516411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:13.516374 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 09:17:13.516615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:13.516447 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 09:17:14.516808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:14.516761 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 09:17:14.517260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:14.516857 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 09:17:15.058486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:15.058447 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" path="/var/lib/kubelet/pods/7fe63c5e-6364-43ed-9d85-781e4f30ab3a/volumes" Nov 01 09:17:15.486369 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:17:15.486326 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:17:16.184290 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:16.184252 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:16.184816 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:16.184603 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:16.185482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:16.185464 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:17:16.993744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:16.993697 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:17:22.311520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.311485 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt"] Nov 01 09:17:22.312345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.312326 5710 cpu_manager.go:414] "containerMap: removing container" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" Nov 01 09:17:22.312429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.312348 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" Nov 01 09:17:22.312429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.312374 5710 cpu_manager.go:414] "containerMap: removing container" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" Nov 01 09:17:22.312429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.312380 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" Nov 01 09:17:22.312528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.312488 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" Nov 01 09:17:22.316130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.316107 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.359753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.359684 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.359937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.359828 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7sg\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.359937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.359880 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.359937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.359922 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.360155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.360000 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.360155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.360084 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.360155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.360134 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.360320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.360181 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.360320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.360298 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.461852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.461808 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.461927 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.461978 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7sg\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462026 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462077 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462104 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462152 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462200 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462236 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462376 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.462875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.462846 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.463061 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.463037 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.463342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.463309 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.464319 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.464296 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.465761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.465737 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.468428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.468399 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.565616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.561680 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt"] Nov 01 09:17:22.575813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.575671 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7sg\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.577852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.577340 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token\") pod \"e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:22.632611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:22.632585 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:23.107064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:23.107015 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt"] Nov 01 09:17:23.108707 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:17:23.108675 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a9ac32_c025_4ab8_8387_89f8bcdeedef.slice/crio-b783cb25de63ca9168e6d44b368779b2a2eca18bf9aa9af2d2fe773e3cc8a72a WatchSource:0}: Error finding container b783cb25de63ca9168e6d44b368779b2a2eca18bf9aa9af2d2fe773e3cc8a72a: Status 404 returned error can't find the container with id b783cb25de63ca9168e6d44b368779b2a2eca18bf9aa9af2d2fe773e3cc8a72a Nov 01 09:17:23.112841 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:23.112804 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:23.112934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:23.112881 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:23.112934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:23.112910 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:24.121834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:24.121794 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" event={"ID":"35a9ac32-c025-4ab8-8387-89f8bcdeedef","Type":"ContainerStarted","Data":"a61e73004f5ca18fb995adcfd03b9a8d1c9d23d2e25c8dea85d1e3b63726eb47"} Nov 01 09:17:24.122331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:24.121856 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" event={"ID":"35a9ac32-c025-4ab8-8387-89f8bcdeedef","Type":"ContainerStarted","Data":"b783cb25de63ca9168e6d44b368779b2a2eca18bf9aa9af2d2fe773e3cc8a72a"} Nov 01 09:17:24.328405 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:24.328274 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podStartSLOduration=2.3282521210000002 podStartE2EDuration="2.328252121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:24.28520499 +0000 UTC m=+2667.373541664" watchObservedRunningTime="2026-11-01 09:17:24.328252121 +0000 UTC m=+2667.416588793" Nov 01 09:17:24.633177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:24.633145 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:24.634951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:24.634921 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:24.635106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:24.634981 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:25.633523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:25.633484 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:25.634079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:25.633547 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:26.633615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:26.633582 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:26.634121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:26.634092 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:27.633766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:27.633709 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:27.634192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:27.633781 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:28.633302 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:28.633251 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:28.643187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:28.633319 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:29.156676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.156638 5710 generic.go:356] "Generic (PLEG): container finished" podID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerID="3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f" exitCode=0 Nov 01 09:17:29.156896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.156706 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" event={"ID":"930bec2c-702a-4b9f-aa04-c1afb3072c24","Type":"ContainerDied","Data":"3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f"} Nov 01 09:17:29.157166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.157152 5710 scope.go:122] "RemoveContainer" containerID="3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f" Nov 01 09:17:29.158700 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.158674 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:29.158776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.158766 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:29.158813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.158801 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:17:29.634014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.633978 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:29.634209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:29.634046 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:30.165783 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:30.165736 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" event={"ID":"930bec2c-702a-4b9f-aa04-c1afb3072c24","Type":"ContainerStarted","Data":"8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7"} Nov 01 09:17:30.492806 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:30.492153 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:17:30.492806 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:30.492290 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 09:17:30.492806 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:30.492333 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 09:17:30.633771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:30.633699 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 09:17:30.633983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:30.633794 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 09:17:31.496745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:31.494192 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 09:17:31.496745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:31.494265 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 09:17:31.638667 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:31.638634 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:31.638998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:31.638972 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:31.639745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:31.639696 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:17:32.497747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:32.495233 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 09:17:32.497747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:32.495325 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 09:17:33.502745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:33.502222 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 09:17:33.502745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:33.502311 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 09:17:34.502747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:34.500130 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 09:17:34.502747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:34.500211 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 09:17:35.551674 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:35.551640 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:17:35.552243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:35.552109 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:17:35.553254 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:35.553230 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:17:45.502438 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:17:45.502389 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:17:46.966871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:46.946889 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:17:47.154634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:47.154591 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:17:47.234808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:47.233847 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:17:47.234808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:47.233942 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:17:47.706746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:47.705878 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:17:47.706746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:47.705950 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:17:55.458627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.458589 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb"] Nov 01 09:17:55.459328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.459311 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe63c5e-6364-43ed-9d85-781e4f30ab3a" containerName="istio-proxy" Nov 01 09:17:55.462898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.462879 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.501653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.501612 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zx9\" (UniqueName: \"kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9\") pod \"payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.501877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.501674 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.559795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.559757 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3b7524\"" Nov 01 09:17:55.560021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.559992 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3b7524-dockercfg-8vgr7\"" Nov 01 09:17:55.562187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.562161 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb"] Nov 01 09:17:55.606592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.606542 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zx9\" (UniqueName: \"kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9\") pod \"payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.606906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.606635 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.607857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.607809 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.621762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.621681 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd"] Nov 01 09:17:55.627495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.627453 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.668809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.668775 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zx9\" (UniqueName: \"kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9\") pod \"payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.678408 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.678374 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd"] Nov 01 09:17:55.778565 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.778467 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:55.809886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.809854 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.810071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.809921 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5js4\" (UniqueName: \"kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4\") pod \"payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.911460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.911416 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.911709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.911484 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5js4\" (UniqueName: \"kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4\") pod \"payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.912285 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.912257 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.969971 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.969940 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5js4\" (UniqueName: \"kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4\") pod \"payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:55.990906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:55.990866 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:56.156539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.156496 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb"] Nov 01 09:17:56.164010 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:17:56.163976 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaac034_4965_4f15_8a7a_9a572cd7a5f8.slice/crio-ae22e3614826c5c0b24db35cd1fada543c7d6b6aeca8c57f38124de1c6d45d19 WatchSource:0}: Error finding container ae22e3614826c5c0b24db35cd1fada543c7d6b6aeca8c57f38124de1c6d45d19: Status 404 returned error can't find the container with id ae22e3614826c5c0b24db35cd1fada543c7d6b6aeca8c57f38124de1c6d45d19 Nov 01 09:17:56.370545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.370501 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd"] Nov 01 09:17:56.378974 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:17:56.378941 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdb0386_ef38_460e_847c_0aa3389ff2db.slice/crio-e5566d37f01869d973bb3f429fc61cd17c10c98382965d73cc917806d8662113 WatchSource:0}: Error finding container e5566d37f01869d973bb3f429fc61cd17c10c98382965d73cc917806d8662113: Status 404 returned error can't find the container with id e5566d37f01869d973bb3f429fc61cd17c10c98382965d73cc917806d8662113 Nov 01 09:17:56.405288 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.405255 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" event={"ID":"cbaac034-4965-4f15-8a7a-9a572cd7a5f8","Type":"ContainerStarted","Data":"b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7"} Nov 01 09:17:56.405440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.405308 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" event={"ID":"cbaac034-4965-4f15-8a7a-9a572cd7a5f8","Type":"ContainerStarted","Data":"ae22e3614826c5c0b24db35cd1fada543c7d6b6aeca8c57f38124de1c6d45d19"} Nov 01 09:17:56.405492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.405437 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:17:56.406700 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.406679 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" event={"ID":"8cdb0386-ef38-460e-847c-0aa3389ff2db","Type":"ContainerStarted","Data":"e5566d37f01869d973bb3f429fc61cd17c10c98382965d73cc917806d8662113"} Nov 01 09:17:56.518669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.518598 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" podStartSLOduration=1.5185800280000001 podStartE2EDuration="1.518580028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:56.517791558 +0000 UTC m=+2699.606128230" watchObservedRunningTime="2026-11-01 09:17:56.518580028 +0000 UTC m=+2699.606916699" Nov 01 09:17:56.967043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.967001 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp"] Nov 01 09:17:56.967266 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:56.967224 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" podUID="137d3e78-18a3-4e77-93a2-33baa71eabaa" containerName="maas-api" containerID="cri-o://ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4" gracePeriod=30 Nov 01 09:17:57.295247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.295220 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:17:57.417328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.417292 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" event={"ID":"8cdb0386-ef38-460e-847c-0aa3389ff2db","Type":"ContainerStarted","Data":"8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960"} Nov 01 09:17:57.417511 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.417469 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:17:57.419115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.419081 5710 generic.go:356] "Generic (PLEG): container finished" podID="137d3e78-18a3-4e77-93a2-33baa71eabaa" containerID="ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4" exitCode=0 Nov 01 09:17:57.419297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.419140 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" Nov 01 09:17:57.419297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.419216 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" event={"ID":"137d3e78-18a3-4e77-93a2-33baa71eabaa","Type":"ContainerDied","Data":"ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4"} Nov 01 09:17:57.419297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.419276 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" event={"ID":"137d3e78-18a3-4e77-93a2-33baa71eabaa","Type":"ContainerDied","Data":"06212cb1899c9053083126eba61f63108b357be8edfea13a5028b98c3c5b37f3"} Nov 01 09:17:57.419474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.419305 5710 scope.go:122] "RemoveContainer" containerID="ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4" Nov 01 09:17:57.431953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.431926 5710 scope.go:122] "RemoveContainer" containerID="ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4" Nov 01 09:17:57.432381 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:17:57.432345 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4\": container with ID starting with ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4 not found: ID does not exist" containerID="ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4" Nov 01 09:17:57.432511 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.432393 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4"} err="failed to get container status \"ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4\": rpc error: code = NotFound desc = could not find container \"ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4\": container with ID starting with ccf770cc20890e54cb63e8d72bbc40666a9cabf46eb30b2165b70b607b5198d4 not found: ID does not exist" Nov 01 09:17:57.449831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.449787 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf\" (UniqueName: \"kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf\") pod \"137d3e78-18a3-4e77-93a2-33baa71eabaa\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " Nov 01 09:17:57.450036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.449867 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\") pod \"137d3e78-18a3-4e77-93a2-33baa71eabaa\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " Nov 01 09:17:57.450036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.449904 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca\") pod \"137d3e78-18a3-4e77-93a2-33baa71eabaa\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " Nov 01 09:17:57.450036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.449939 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs\") pod \"137d3e78-18a3-4e77-93a2-33baa71eabaa\" (UID: \"137d3e78-18a3-4e77-93a2-33baa71eabaa\") " Nov 01 09:17:57.450565 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.450532 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca" pod "137d3e78-18a3-4e77-93a2-33baa71eabaa" (UID: "137d3e78-18a3-4e77-93a2-33baa71eabaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:17:57.452345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.452317 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls" pod "137d3e78-18a3-4e77-93a2-33baa71eabaa" (UID: "137d3e78-18a3-4e77-93a2-33baa71eabaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:17:57.452479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.452458 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf" pod "137d3e78-18a3-4e77-93a2-33baa71eabaa" (UID: "137d3e78-18a3-4e77-93a2-33baa71eabaa"). InnerVolumeSpecName "kube-api-access-mm5qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:17:57.452752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.452684 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs" pod "137d3e78-18a3-4e77-93a2-33baa71eabaa" (UID: "137d3e78-18a3-4e77-93a2-33baa71eabaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:17:57.545636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.545511 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" podStartSLOduration=2.545489198 podStartE2EDuration="2.545489198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:17:57.545467776 +0000 UTC m=+2700.633804449" watchObservedRunningTime="2026-11-01 09:17:57.545489198 +0000 UTC m=+2700.633825871" Nov 01 09:17:57.553445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.552168 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm5qf\" (UniqueName: \"kubernetes.io/projected/137d3e78-18a3-4e77-93a2-33baa71eabaa-kube-api-access-mm5qf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:57.553445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.552207 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:57.553445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.552223 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137d3e78-18a3-4e77-93a2-33baa71eabaa-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:57.553445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.552241 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137d3e78-18a3-4e77-93a2-33baa71eabaa-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:17:57.832898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.832807 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp"] Nov 01 09:17:57.836868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:57.836839 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp"] Nov 01 09:17:59.058472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:17:59.058436 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137d3e78-18a3-4e77-93a2-33baa71eabaa" path="/var/lib/kubelet/pods/137d3e78-18a3-4e77-93a2-33baa71eabaa/volumes" Nov 01 09:18:00.233558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:00.233490 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1c173-79d6686bcf-c9fmp" podUID="137d3e78-18a3-4e77-93a2-33baa71eabaa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.178:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 09:18:01.001870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:01.001842 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:18:01.016706 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:01.016681 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:18:07.422452 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.422421 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:18:07.772828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.772734 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt"] Nov 01 09:18:07.773349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.773335 5710 cpu_manager.go:414] "containerMap: removing container" podUID="137d3e78-18a3-4e77-93a2-33baa71eabaa" containerName="maas-api" Nov 01 09:18:07.773395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.773351 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137d3e78-18a3-4e77-93a2-33baa71eabaa" containerName="maas-api" Nov 01 09:18:07.773617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.773589 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="137d3e78-18a3-4e77-93a2-33baa71eabaa" containerName="maas-api" Nov 01 09:18:07.778960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.778935 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.835098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.835054 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt"] Nov 01 09:18:07.853956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.853919 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.853978 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854134 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854179 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854313 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854371 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854421 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7bp\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854455 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.854609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.854504 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.955939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.955893 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956023 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956091 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956141 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956289 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956266 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956319 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7bp\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956352 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956381 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956412 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956473 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956497 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956787 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956763 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.956920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.956837 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.957352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.957328 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.958764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.958700 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:07.959005 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:07.958982 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:08.014163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.014128 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:08.014331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.014238 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7bp\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:08.094436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.094400 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:08.430277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.430190 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:18:08.528702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.528668 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt"] Nov 01 09:18:08.535676 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:18:08.535648 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e899f4_3ccc_4676_b68b_f9272cedb9b8.slice/crio-15b9d4d4e7e110b67c3603058bd313f3be37948801af2c571c7eb88c1ea73eb3 WatchSource:0}: Error finding container 15b9d4d4e7e110b67c3603058bd313f3be37948801af2c571c7eb88c1ea73eb3: Status 404 returned error can't find the container with id 15b9d4d4e7e110b67c3603058bd313f3be37948801af2c571c7eb88c1ea73eb3 Nov 01 09:18:08.538937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.538903 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:18:08.539059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.538969 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:18:08.539059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:08.538997 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:18:09.512709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:09.512666 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" event={"ID":"f5e899f4-3ccc-4676-b68b-f9272cedb9b8","Type":"ContainerStarted","Data":"6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905"} Nov 01 09:18:09.513166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:09.512774 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" event={"ID":"f5e899f4-3ccc-4676-b68b-f9272cedb9b8","Type":"ContainerStarted","Data":"15b9d4d4e7e110b67c3603058bd313f3be37948801af2c571c7eb88c1ea73eb3"} Nov 01 09:18:09.626995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:09.626936 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podStartSLOduration=2.626922356 podStartE2EDuration="2.626922356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:09.625860137 +0000 UTC m=+2712.714196808" watchObservedRunningTime="2026-11-01 09:18:09.626922356 +0000 UTC m=+2712.715259028" Nov 01 09:18:10.105782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:10.105131 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:10.107920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:10.107885 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:10.108093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:10.107943 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:11.103749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:11.103312 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:11.103749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:11.103378 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:11.877181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:11.877142 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:18:12.096745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:12.096265 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:12.096745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:12.096342 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:13.098745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:13.098212 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:13.098745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:13.098284 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:14.096749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:14.096248 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:14.096749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:14.096338 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:15.097328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:15.097273 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:15.098101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:15.098047 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:15.435879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:15.435753 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 09:18:15.496308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:15.495873 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 09:18:15.594120 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:18:15.593013 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:18:16.095225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:16.095178 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Nov 01 09:18:16.095466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:16.095245 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Nov 01 09:18:18.136908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:18.136863 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:18:18.137524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:18.136961 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:18:18.943837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:18.943801 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:18.944243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:18.944213 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:18.950022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:18.949934 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 09:18:38.015251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.015214 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r"] Nov 01 09:18:38.021275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.021243 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071536 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071624 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071709 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071816 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071846 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071927 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.071979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071960 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.072123 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.071996 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lhf\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.072123 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.072079 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.172809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.172768 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.172830 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.172881 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.172908 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.172953 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.172990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173032 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lhf\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173099 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173185 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173441 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173465 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173559 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.173930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.173901 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.174358 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.174332 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.175817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.175793 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.175963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.175936 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.217566 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.217526 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r"] Nov 01 09:18:38.232179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.232145 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.232362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.232276 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lhf\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf\") pod \"e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.337262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.337156 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:38.536106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.536072 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm"] Nov 01 09:18:38.540784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.540750 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.578874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.578838 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.579077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.578975 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.579077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.579025 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlkd\" (UniqueName: \"kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.579198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.579184 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.589664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.589579 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3b7524-serving-cert\"" Nov 01 09:18:38.597526 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.597493 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm"] Nov 01 09:18:38.680513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.680474 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.680763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.680569 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.680763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.680680 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.680990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.680768 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlkd\" (UniqueName: \"kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.680990 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:18:38.680842 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-serving-cert: secret "maas-api-e2e-praxis-b-3b7524-serving-cert" not found Nov 01 09:18:38.680990 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:18:38.680921 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls podName:a5419681-84fc-4716-ab19-16908093c3a4 nodeName:}" failed. No retries permitted until 2026-11-01 09:18:39.180897828 +0000 UTC m=+2742.269234485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls") pod "maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" (UID: "a5419681-84fc-4716-ab19-16908093c3a4") : secret "maas-api-e2e-praxis-b-3b7524-serving-cert" not found Nov 01 09:18:38.681448 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.681418 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.683656 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.683631 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.738973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.738939 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlkd\" (UniqueName: \"kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:38.826606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.826579 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r"] Nov 01 09:18:38.830260 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:18:38.830226 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fa8130_1812_40fe_82ff_5aa990333199.slice/crio-a392737dcafaca501779f554e230b98c8245d73a0b1ab70dcf31b46b73ff1f92 WatchSource:0}: Error finding container a392737dcafaca501779f554e230b98c8245d73a0b1ab70dcf31b46b73ff1f92: Status 404 returned error can't find the container with id a392737dcafaca501779f554e230b98c8245d73a0b1ab70dcf31b46b73ff1f92 Nov 01 09:18:38.833183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.833149 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:18:38.833293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.833221 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:18:38.833293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.833254 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:18:38.897870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:38.897827 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" event={"ID":"c3fa8130-1812-40fe-82ff-5aa990333199","Type":"ContainerStarted","Data":"a392737dcafaca501779f554e230b98c8245d73a0b1ab70dcf31b46b73ff1f92"} Nov 01 09:18:39.209867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:39.209773 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:39.213035 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:39.213000 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:39.456097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:39.456060 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:39.847604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:39.847572 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm"] Nov 01 09:18:39.854887 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:18:39.854843 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5419681_84fc_4716_ab19_16908093c3a4.slice/crio-9017a6f20ecd384febd07d3d7a520e1de8c1754471b958080a704c487de7f6b9 WatchSource:0}: Error finding container 9017a6f20ecd384febd07d3d7a520e1de8c1754471b958080a704c487de7f6b9: Status 404 returned error can't find the container with id 9017a6f20ecd384febd07d3d7a520e1de8c1754471b958080a704c487de7f6b9 Nov 01 09:18:39.916163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:39.916107 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" event={"ID":"c3fa8130-1812-40fe-82ff-5aa990333199","Type":"ContainerStarted","Data":"a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2"} Nov 01 09:18:39.917608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:39.917575 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" event={"ID":"a5419681-84fc-4716-ab19-16908093c3a4","Type":"ContainerStarted","Data":"9017a6f20ecd384febd07d3d7a520e1de8c1754471b958080a704c487de7f6b9"} Nov 01 09:18:40.029646 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:40.029589 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podStartSLOduration=3.029576237 podStartE2EDuration="3.029576237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:40.029030273 +0000 UTC m=+2743.117366956" watchObservedRunningTime="2026-11-01 09:18:40.029576237 +0000 UTC m=+2743.117912909" Nov 01 09:18:40.337505 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:40.337418 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:40.339491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:40.339459 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:40.339628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:40.339509 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:41.338690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:41.338595 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:41.338690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:41.338661 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:41.440795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:41.440756 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:18:41.933045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:41.933005 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" event={"ID":"a5419681-84fc-4716-ab19-16908093c3a4","Type":"ContainerStarted","Data":"af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913"} Nov 01 09:18:41.933280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:41.933225 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:42.063755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:42.063559 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" podStartSLOduration=2.964093121 podStartE2EDuration="4.06353663s" totalImagesPullingTime="1.099443509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:42.05027389 +0000 UTC m=+2745.138610566" watchObservedRunningTime="2026-11-01 09:18:42.06353663 +0000 UTC m=+2745.151873303" Nov 01 09:18:42.342660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:42.342166 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:42.342660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:42.342234 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:43.340753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:43.340433 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:43.340753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:43.340553 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:44.340745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:44.340032 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:44.340745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:44.340100 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:44.487744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:44.486831 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 09:18:45.352150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:45.338150 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:45.352150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:45.338234 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:45.842950 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:18:45.842910 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:18:46.337895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:46.337781 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:46.337895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:46.337851 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:47.338556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:47.338517 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 09:18:47.339118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:47.338593 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 09:18:47.968171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:47.968128 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:18:49.378939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:49.378891 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:18:49.379439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:49.378968 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:18:49.951581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:49.951535 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:49.952040 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:49.952015 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:49.953410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:49.953383 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:18:54.403525 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.403464 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8"] Nov 01 09:18:54.409229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.409202 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.464151 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.464093 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8"] Nov 01 09:18:54.492614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.492552 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-035625\"" Nov 01 09:18:54.492866 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.492643 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-035625-dockercfg-cfhl4\"" Nov 01 09:18:54.549174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.549135 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5"] Nov 01 09:18:54.553635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.553606 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.564053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.564019 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.564221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.564131 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dv5l\" (UniqueName: \"kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l\") pod \"payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.607665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.607630 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5"] Nov 01 09:18:54.665886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.665787 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.665886 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.665853 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.666102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.665953 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dg82\" (UniqueName: \"kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82\") pod \"payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.666102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.666008 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dv5l\" (UniqueName: \"kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l\") pod \"payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.666580 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.666561 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.726804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.726706 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dv5l\" (UniqueName: \"kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l\") pod \"payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.755494 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.755454 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:54.767918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.767881 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.768058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.767994 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dg82\" (UniqueName: \"kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82\") pod \"payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.768751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.768709 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.835868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.835562 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dg82\" (UniqueName: \"kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82\") pod \"payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:54.868595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:54.868562 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:55.134439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:55.134400 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8"] Nov 01 09:18:55.259375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:55.259341 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5"] Nov 01 09:18:55.266752 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:18:55.266692 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f306c2_bbbc_41bf_8628_5cb0f1998b11.slice/crio-0cc0bea14e3b6bfc6aa394a62e3e8afdee5a24521bb367ebfee74913147c9935 WatchSource:0}: Error finding container 0cc0bea14e3b6bfc6aa394a62e3e8afdee5a24521bb367ebfee74913147c9935: Status 404 returned error can't find the container with id 0cc0bea14e3b6bfc6aa394a62e3e8afdee5a24521bb367ebfee74913147c9935 Nov 01 09:18:55.271673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:55.271648 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" event={"ID":"919c821a-75eb-4d38-8722-5e7d19d5fd8d","Type":"ContainerStarted","Data":"79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772"} Nov 01 09:18:55.271874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:55.271708 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" event={"ID":"919c821a-75eb-4d38-8722-5e7d19d5fd8d","Type":"ContainerStarted","Data":"b4568c2c0df7970bcdb6e3a0c7704974081a1a4046095c54b2bd2edc16f36c92"} Nov 01 09:18:55.271874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:55.271762 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:18:55.381424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:55.381362 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" podStartSLOduration=1.381347187 podStartE2EDuration="1.381347187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:55.380632821 +0000 UTC m=+2758.468969492" watchObservedRunningTime="2026-11-01 09:18:55.381347187 +0000 UTC m=+2758.469683894" Nov 01 09:18:56.293846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:56.293809 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" event={"ID":"f3f306c2-bbbc-41bf-8628-5cb0f1998b11","Type":"ContainerStarted","Data":"3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309"} Nov 01 09:18:56.294319 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:56.293871 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" event={"ID":"f3f306c2-bbbc-41bf-8628-5cb0f1998b11","Type":"ContainerStarted","Data":"0cc0bea14e3b6bfc6aa394a62e3e8afdee5a24521bb367ebfee74913147c9935"} Nov 01 09:18:56.294319 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:56.294055 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:18:56.402432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:18:56.402367 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" podStartSLOduration=2.402349316 podStartE2EDuration="2.402349316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:18:56.40181231 +0000 UTC m=+2759.490148982" watchObservedRunningTime="2026-11-01 09:18:56.402349316 +0000 UTC m=+2759.490685989" Nov 01 09:19:04.517809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.517764 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc"] Nov 01 09:19:04.524737 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.524684 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.574454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.574407 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-fd5a10-openshift-default-dockercfg-r6v87\"" Nov 01 09:19:04.618688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.618643 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944fr\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.618981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.618761 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.618981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.618807 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.618981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.618942 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.619180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.618988 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.619180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.619032 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.619180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.619095 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.619180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.619126 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.619180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.619159 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.623782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.623751 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc"] Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.723623 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944fr\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.723740 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.723785 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.723915 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.723965 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.724075 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.724182 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.724270 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.724313 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.732671 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.724924 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.735267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.734486 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.735451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.735277 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.739222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.739167 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.740816 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.739463 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.756269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.752565 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.803848 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.803494 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944fr\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.818394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.809040 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.845865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.819262 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token\") pod \"e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:04.847074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:04.847038 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:05.297898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:05.297860 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc"] Nov 01 09:19:05.304451 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:19:05.304419 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded492f1a_d299_43d0_a059_af6d93321d0e.slice/crio-c7870563aa78c447c5de37cf2d398d417b8fe65253ce4849e69f699423873c32 WatchSource:0}: Error finding container c7870563aa78c447c5de37cf2d398d417b8fe65253ce4849e69f699423873c32: Status 404 returned error can't find the container with id c7870563aa78c447c5de37cf2d398d417b8fe65253ce4849e69f699423873c32 Nov 01 09:19:05.307024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:05.306990 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:05.307187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:05.307069 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:05.307187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:05.307096 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:05.356081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:05.356047 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" event={"ID":"ed492f1a-d299-43d0-a059-af6d93321d0e","Type":"ContainerStarted","Data":"c7870563aa78c447c5de37cf2d398d417b8fe65253ce4849e69f699423873c32"} Nov 01 09:19:06.297420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:06.297387 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:19:06.363812 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:06.363759 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" event={"ID":"ed492f1a-d299-43d0-a059-af6d93321d0e","Type":"ContainerStarted","Data":"d40e5ba0cd09d18a472146ea2184ab0b7cda6415774f8412057c5284cb5e6cb3"} Nov 01 09:19:06.532701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:06.532623 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podStartSLOduration=2.532601016 podStartE2EDuration="2.532601016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:06.528325048 +0000 UTC m=+2769.616661744" watchObservedRunningTime="2026-11-01 09:19:06.532601016 +0000 UTC m=+2769.620937681" Nov 01 09:19:06.847910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:06.847869 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:06.854055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:06.854025 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:07.308773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:07.308676 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:19:07.370637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:07.370608 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:07.371882 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:07.371855 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:19:11.529954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:11.529879 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 09:19:11.790952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:11.790832 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" podUID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 09:19:13.568128 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.568084 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp"] Nov 01 09:19:13.574077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.574048 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:13.649747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.643963 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-fd5a10\"" Nov 01 09:19:13.651081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.651017 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp"] Nov 01 09:19:13.703742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.703383 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-fd5a10-dockercfg-qb8sr\"" Nov 01 09:19:13.761198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.761088 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jjh\" (UniqueName: \"kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh\") pod \"payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:13.761455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.761413 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:13.863124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.863086 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jjh\" (UniqueName: \"kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh\") pod \"payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:13.863343 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.863248 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:13.864085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.864065 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:13.921177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:13.921134 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jjh\" (UniqueName: \"kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh\") pod \"payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:14.191482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.191389 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:14.337106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.336697 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8"] Nov 01 09:19:14.343990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.343949 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.398299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.398264 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfdd13-openshift-default-dockercfg-jf87f\"" Nov 01 09:19:14.443964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.443870 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8"] Nov 01 09:19:14.476461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476418 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.476651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476527 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.476651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476566 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.476651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476590 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.476651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476620 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.476890 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476808 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45bq\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.476935 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476916 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.477012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.476996 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.477149 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.477130 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.574142 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.574097 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp"] Nov 01 09:19:14.578169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578137 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45bq\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578287 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578229 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578330 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578426 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578542 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578571 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578595 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578619 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.578708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.578656 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.579164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.579136 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.579466 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.579445 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.579556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.579532 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.580124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.580084 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.581358 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.581329 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.581692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.581667 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.582140 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:19:14.582112 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode555ea93_40d7_44ca_82b5_be5e1d2624c6.slice/crio-bc7735bbb28213d7c173049452520e4a9094c3a10da1294ebaab818aa7656a70 WatchSource:0}: Error finding container bc7735bbb28213d7c173049452520e4a9094c3a10da1294ebaab818aa7656a70: Status 404 returned error can't find the container with id bc7735bbb28213d7c173049452520e4a9094c3a10da1294ebaab818aa7656a70 Nov 01 09:19:14.585208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.582799 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.636032 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.635987 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45bq\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.636183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.636112 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.668199 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.668174 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:14.755054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.754763 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" event={"ID":"e555ea93-40d7-44ca-82b5-be5e1d2624c6","Type":"ContainerStarted","Data":"73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80"} Nov 01 09:19:14.755054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.754840 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:14.755054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.754864 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" event={"ID":"e555ea93-40d7-44ca-82b5-be5e1d2624c6","Type":"ContainerStarted","Data":"bc7735bbb28213d7c173049452520e4a9094c3a10da1294ebaab818aa7656a70"} Nov 01 09:19:14.863422 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:14.863350 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" podStartSLOduration=1.863331064 podStartE2EDuration="1.863331064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:14.862926127 +0000 UTC m=+2777.951262800" watchObservedRunningTime="2026-11-01 09:19:14.863331064 +0000 UTC m=+2777.951667737" Nov 01 09:19:15.115309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.115264 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8"] Nov 01 09:19:15.122054 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:19:15.122005 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131e4285_dd5e_4eea_9449_596a1aac2c21.slice/crio-1803e029c733b00d7aca6cb91b3b2803b7c7e9609e2e9947b8eeeacddd0fa045 WatchSource:0}: Error finding container 1803e029c733b00d7aca6cb91b3b2803b7c7e9609e2e9947b8eeeacddd0fa045: Status 404 returned error can't find the container with id 1803e029c733b00d7aca6cb91b3b2803b7c7e9609e2e9947b8eeeacddd0fa045 Nov 01 09:19:15.125024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.124985 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:15.125158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.125080 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:15.125158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.125124 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:15.479909 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:19:15.479789 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:19:15.763349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.763255 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" event={"ID":"131e4285-dd5e-4eea-9449-596a1aac2c21","Type":"ContainerStarted","Data":"393eb08c095a2d51c5d517c860c268ae331b1e3fd4dc0ab3bcf625b37532f9e1"} Nov 01 09:19:15.763349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.763342 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" event={"ID":"131e4285-dd5e-4eea-9449-596a1aac2c21","Type":"ContainerStarted","Data":"1803e029c733b00d7aca6cb91b3b2803b7c7e9609e2e9947b8eeeacddd0fa045"} Nov 01 09:19:15.877138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:15.877070 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podStartSLOduration=1.8770542639999999 podStartE2EDuration="1.877054264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:15.87633815 +0000 UTC m=+2778.964674838" watchObservedRunningTime="2026-11-01 09:19:15.877054264 +0000 UTC m=+2778.965390935" Nov 01 09:19:16.668802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:16.668756 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:16.670414 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:16.670383 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:19:16.670587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:16.670443 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:19:17.673791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:17.669856 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:19:17.673791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:17.669936 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:19:18.668727 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:18.668688 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:19:18.668941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:18.668810 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:19:19.668664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:19.668623 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:19:19.669173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:19.668681 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:19:19.927181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:19.927067 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r"] Nov 01 09:19:20.669321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:20.669274 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:19:20.669910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:20.669356 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:19:21.669070 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:21.669029 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:19:21.669291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:21.669105 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:19:23.180924 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:23.180889 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:23.181964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:23.181673 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:23.182875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:23.182841 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:19:24.895978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:24.895939 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:19:25.552666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.552624 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq"] Nov 01 09:19:25.557271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.557218 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.653049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.653000 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6752e4-openshift-default-dockercfg-z6rpx\"" Nov 01 09:19:25.656992 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.656956 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq"] Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672115 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672228 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672302 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672418 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672478 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672652 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672700 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7ss\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672774 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.685564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.672902 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.771783 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.767772 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774553 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774607 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774644 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774671 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774776 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774805 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774888 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774908 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7ss\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.774937 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.775622 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.775934 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.776235 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.776780 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.777223 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.795743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.786288 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.803815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.799616 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.834427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.834396 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.838826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.838692 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7ss\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss\") pod \"e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:25.874763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:25.874708 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:26.353765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:26.353701 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq"] Nov 01 09:19:26.359171 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:19:26.359140 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200c3913_22d9_4f43_93cd_03ac77c8f7de.slice/crio-8ea7139f778418dcf3f64b48eeecadfb202ecaf1fadeaa78f94c6d46f1cb80fe WatchSource:0}: Error finding container 8ea7139f778418dcf3f64b48eeecadfb202ecaf1fadeaa78f94c6d46f1cb80fe: Status 404 returned error can't find the container with id 8ea7139f778418dcf3f64b48eeecadfb202ecaf1fadeaa78f94c6d46f1cb80fe Nov 01 09:19:26.362862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:26.362804 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:26.362951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:26.362921 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:26.362995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:26.362970 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:19:26.877814 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:26.877769 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" event={"ID":"200c3913-22d9-4f43-93cd-03ac77c8f7de","Type":"ContainerStarted","Data":"6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56"} Nov 01 09:19:26.878039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:26.877844 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" event={"ID":"200c3913-22d9-4f43-93cd-03ac77c8f7de","Type":"ContainerStarted","Data":"8ea7139f778418dcf3f64b48eeecadfb202ecaf1fadeaa78f94c6d46f1cb80fe"} Nov 01 09:19:27.875961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:27.875923 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:27.877728 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:27.877694 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 09:19:27.877815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:27.877773 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 09:19:28.875545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:28.875504 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 09:19:28.875998 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:28.875944 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 09:19:29.876199 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:29.876167 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 09:19:29.876615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:29.876254 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 09:19:30.875608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:30.875562 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 09:19:30.875862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:30.875622 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 09:19:31.875994 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:31.875957 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 09:19:31.876428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:31.876015 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 09:19:32.876101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:32.876055 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 09:19:32.876530 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:32.876132 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 09:19:33.880213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:33.880181 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:33.880676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:33.880652 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:33.881428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:33.881408 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:19:34.041539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:34.041477 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podStartSLOduration=9.041460833 podStartE2EDuration="9.041460833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:27.004995324 +0000 UTC m=+2790.093331999" watchObservedRunningTime="2026-11-01 09:19:34.041460833 +0000 UTC m=+2797.129797505" Nov 01 09:19:34.950187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:34.950144 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" podUID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" containerName="main" containerID="cri-o://d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb" gracePeriod=45 Nov 01 09:19:35.233253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.233227 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:19:35.291599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291559 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj\" (UniqueName: \"kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj\") pod \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " Nov 01 09:19:35.291845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291623 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm\") pod \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " Nov 01 09:19:35.291845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291646 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache\") pod \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " Nov 01 09:19:35.291845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291695 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home\") pod \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " Nov 01 09:19:35.291845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291776 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs\") pod \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " Nov 01 09:19:35.291845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291835 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir\") pod \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\" (UID: \"b601ac90-0ffc-4bdd-9b32-9f10356af28d\") " Nov 01 09:19:35.292148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.291988 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache" pod "b601ac90-0ffc-4bdd-9b32-9f10356af28d" (UID: "b601ac90-0ffc-4bdd-9b32-9f10356af28d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:19:35.292148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.292005 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home" pod "b601ac90-0ffc-4bdd-9b32-9f10356af28d" (UID: "b601ac90-0ffc-4bdd-9b32-9f10356af28d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:19:35.292237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.292214 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir" pod "b601ac90-0ffc-4bdd-9b32-9f10356af28d" (UID: "b601ac90-0ffc-4bdd-9b32-9f10356af28d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:19:35.292619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.292594 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:19:35.292771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.292622 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:19:35.292771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.292640 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:19:35.294161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.294141 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm" pod "b601ac90-0ffc-4bdd-9b32-9f10356af28d" (UID: "b601ac90-0ffc-4bdd-9b32-9f10356af28d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:19:35.294261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.294211 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj" pod "b601ac90-0ffc-4bdd-9b32-9f10356af28d" (UID: "b601ac90-0ffc-4bdd-9b32-9f10356af28d"). InnerVolumeSpecName "kube-api-access-nfqlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:19:35.294261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.294222 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs" pod "b601ac90-0ffc-4bdd-9b32-9f10356af28d" (UID: "b601ac90-0ffc-4bdd-9b32-9f10356af28d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:19:35.393748 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.393676 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfqlj\" (UniqueName: \"kubernetes.io/projected/b601ac90-0ffc-4bdd-9b32-9f10356af28d-kube-api-access-nfqlj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:19:35.393748 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.393742 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b601ac90-0ffc-4bdd-9b32-9f10356af28d-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:19:35.393748 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.393758 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b601ac90-0ffc-4bdd-9b32-9f10356af28d-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:19:35.963539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.963501 5710 generic.go:356] "Generic (PLEG): container finished" podID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" containerID="d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb" exitCode=0 Nov 01 09:19:35.963539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.963548 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" event={"ID":"b601ac90-0ffc-4bdd-9b32-9f10356af28d","Type":"ContainerDied","Data":"d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb"} Nov 01 09:19:35.964174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.963561 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" Nov 01 09:19:35.964174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.963575 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r" event={"ID":"b601ac90-0ffc-4bdd-9b32-9f10356af28d","Type":"ContainerDied","Data":"45c319dabc945eb6f20a9fa989f9d596226dfaa41dc4d61600683c02ecfa323f"} Nov 01 09:19:35.964174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.963592 5710 scope.go:122] "RemoveContainer" containerID="d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb" Nov 01 09:19:35.975132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.975114 5710 scope.go:122] "RemoveContainer" containerID="d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb" Nov 01 09:19:35.975462 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:19:35.975439 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb\": container with ID starting with d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb not found: ID does not exist" containerID="d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb" Nov 01 09:19:35.975539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:35.975477 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb"} err="failed to get container status \"d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb\": rpc error: code = NotFound desc = could not find container \"d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb\": container with ID starting with d7e2e50aecea9f684c6d90e2f5d0b049a53fee7fb509fb69047cf16d63b53eeb not found: ID does not exist" Nov 01 09:19:36.118640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:36.118603 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r"] Nov 01 09:19:36.165089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:36.165051 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-6752e4/praxis-route-6752e4-kserve-7db4b6c98c-shq7r"] Nov 01 09:19:37.070878 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:37.070841 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" path="/var/lib/kubelet/pods/b601ac90-0ffc-4bdd-9b32-9f10356af28d/volumes" Nov 01 09:19:45.481152 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:19:45.481083 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:19:52.469440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.469404 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4"] Nov 01 09:19:52.470222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.470086 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" containerName="main" Nov 01 09:19:52.470222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.470103 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" containerName="main" Nov 01 09:19:52.470332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.470227 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b601ac90-0ffc-4bdd-9b32-9f10356af28d" containerName="main" Nov 01 09:19:52.473790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.473770 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.568024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.567971 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.568248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.568045 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.568248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.568107 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.568409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.568381 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccxh\" (UniqueName: \"kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.570108 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.570083 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a87a1d-serving-cert\"" Nov 01 09:19:52.573174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.573145 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4"] Nov 01 09:19:52.669834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.669796 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccxh\" (UniqueName: \"kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.670064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.669886 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.670064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.669942 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.670064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.669966 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.670268 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:19:52.670131 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-serving-cert: secret "maas-api-e2e-shared-a-w1-a87a1d-serving-cert" not found Nov 01 09:19:52.670268 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:19:52.670192 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls podName:1a24441e-066c-43b0-b4ea-509691a6f607 nodeName:}" failed. No retries permitted until 2026-11-01 09:19:53.170172195 +0000 UTC m=+2816.258508846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls") pod "maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" (UID: "1a24441e-066c-43b0-b4ea-509691a6f607") : secret "maas-api-e2e-shared-a-w1-a87a1d-serving-cert" not found Nov 01 09:19:52.671014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.670988 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.673444 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.673410 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:52.728445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:52.728366 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccxh\" (UniqueName: \"kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:53.199373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:53.199326 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:53.202112 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:53.202044 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:53.394117 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:53.394072 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:53.776972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:53.776878 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4"] Nov 01 09:19:53.783698 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:19:53.783665 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a24441e_066c_43b0_b4ea_509691a6f607.slice/crio-7dd2205b54b3d8c1e6c1e6ac9f7e84ebeda24fd99f787911bc4f2c5e5d2a6794 WatchSource:0}: Error finding container 7dd2205b54b3d8c1e6c1e6ac9f7e84ebeda24fd99f787911bc4f2c5e5d2a6794: Status 404 returned error can't find the container with id 7dd2205b54b3d8c1e6c1e6ac9f7e84ebeda24fd99f787911bc4f2c5e5d2a6794 Nov 01 09:19:54.116008 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:54.115967 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" event={"ID":"1a24441e-066c-43b0-b4ea-509691a6f607","Type":"ContainerStarted","Data":"7dd2205b54b3d8c1e6c1e6ac9f7e84ebeda24fd99f787911bc4f2c5e5d2a6794"} Nov 01 09:19:55.124296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:55.124250 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" event={"ID":"1a24441e-066c-43b0-b4ea-509691a6f607","Type":"ContainerStarted","Data":"f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e"} Nov 01 09:19:55.124786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:55.124321 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:19:55.265363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:19:55.265306 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" podStartSLOduration=2.367182863 podStartE2EDuration="3.265290294s" totalImagesPullingTime="898.107431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:19:55.260579794 +0000 UTC m=+2818.348916467" watchObservedRunningTime="2026-11-01 09:19:55.265290294 +0000 UTC m=+2818.353626966" Nov 01 09:20:00.313502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:00.313463 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr"] Nov 01 09:20:01.137021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:01.136982 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 09:20:05.314462 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:05.314413 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerName="payload-processing" containerID="cri-o://fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1" gracePeriod=25 Nov 01 09:20:07.648795 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:07.648751 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:20:13.132337 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.132291 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb"] Nov 01 09:20:13.138224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.138198 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.188163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.188128 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-fd5a10-serving-cert\"" Nov 01 09:20:13.237383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.237344 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb"] Nov 01 09:20:13.308968 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.308926 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdkwv\" (UniqueName: \"kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.309136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.309006 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.309136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.309055 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.309243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.309158 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.410772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.410641 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.411222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.411175 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.411980 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.411953 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.417369 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:13.411109 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-serving-cert: secret "maas-api-e2e-shared-a-w4-fd5a10-serving-cert" not found Nov 01 09:20:13.422864 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:13.422843 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls podName:6d1ad7a2-34e4-4fd3-9841-a7059bd752bd nodeName:}" failed. No retries permitted until 2026-11-01 09:20:13.922815574 +0000 UTC m=+2837.011152241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls") pod "maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" (UID: "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd") : secret "maas-api-e2e-shared-a-w4-fd5a10-serving-cert" not found Nov 01 09:20:13.423069 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.420452 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdkwv\" (UniqueName: \"kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.423360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.413016 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.432423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.431571 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.478475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.478441 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdkwv\" (UniqueName: \"kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.929419 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.929374 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:13.932183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:13.932158 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:14.052920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:14.052882 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:14.434521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:14.434480 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb"] Nov 01 09:20:14.441934 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:20:14.441901 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1ad7a2_34e4_4fd3_9841_a7059bd752bd.slice/crio-81f449f64fc87b3eed760df22674c4e064e663b09c2d18f62eb72786e9f88aef WatchSource:0}: Error finding container 81f449f64fc87b3eed760df22674c4e064e663b09c2d18f62eb72786e9f88aef: Status 404 returned error can't find the container with id 81f449f64fc87b3eed760df22674c4e064e663b09c2d18f62eb72786e9f88aef Nov 01 09:20:14.834969 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:14.834867 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:20:15.264800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:15.264759 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" event={"ID":"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd","Type":"ContainerStarted","Data":"81f449f64fc87b3eed760df22674c4e064e663b09c2d18f62eb72786e9f88aef"} Nov 01 09:20:15.484755 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:15.484675 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:20:16.310748 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:16.310681 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" event={"ID":"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd","Type":"ContainerStarted","Data":"dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18"} Nov 01 09:20:16.320742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:16.316163 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:16.426943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:16.426881 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" podStartSLOduration=2.482383759 podStartE2EDuration="3.426866133s" totalImagesPullingTime="944.482374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:20:16.425041427 +0000 UTC m=+2839.513378137" watchObservedRunningTime="2026-11-01 09:20:16.426866133 +0000 UTC m=+2839.515202804" Nov 01 09:20:17.648792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:17.648748 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:20:22.598118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.598086 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68"] Nov 01 09:20:22.598570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.598319 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" podUID="25b87df0-e39e-40bc-9500-09807efa8334" containerName="maas-api" containerID="cri-o://f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419" gracePeriod=30 Nov 01 09:20:22.937406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.937374 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:20:22.988013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.981776 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs\") pod \"25b87df0-e39e-40bc-9500-09807efa8334\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " Nov 01 09:20:22.988013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.981897 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca\") pod \"25b87df0-e39e-40bc-9500-09807efa8334\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " Nov 01 09:20:22.988013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.982013 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn\" (UniqueName: \"kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn\") pod \"25b87df0-e39e-40bc-9500-09807efa8334\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " Nov 01 09:20:22.988013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.982059 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\") pod \"25b87df0-e39e-40bc-9500-09807efa8334\" (UID: \"25b87df0-e39e-40bc-9500-09807efa8334\") " Nov 01 09:20:22.998778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.993418 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca" pod "25b87df0-e39e-40bc-9500-09807efa8334" (UID: "25b87df0-e39e-40bc-9500-09807efa8334"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:20:22.998778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:22.998332 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls" pod "25b87df0-e39e-40bc-9500-09807efa8334" (UID: "25b87df0-e39e-40bc-9500-09807efa8334"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:20:23.023742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.023642 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn" pod "25b87df0-e39e-40bc-9500-09807efa8334" (UID: "25b87df0-e39e-40bc-9500-09807efa8334"). InnerVolumeSpecName "kube-api-access-mpddn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:20:23.024086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.024051 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs" pod "25b87df0-e39e-40bc-9500-09807efa8334" (UID: "25b87df0-e39e-40bc-9500-09807efa8334"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:20:23.083933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.083900 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:23.083933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.083929 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b87df0-e39e-40bc-9500-09807efa8334-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:23.083933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.083938 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpddn\" (UniqueName: \"kubernetes.io/projected/25b87df0-e39e-40bc-9500-09807efa8334-kube-api-access-mpddn\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:23.084200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.083949 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b87df0-e39e-40bc-9500-09807efa8334-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:23.329417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.329333 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:20:23.367504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.367469 5710 generic.go:356] "Generic (PLEG): container finished" podID="25b87df0-e39e-40bc-9500-09807efa8334" containerID="f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419" exitCode=0 Nov 01 09:20:23.367760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.367527 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" Nov 01 09:20:23.367760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.367553 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" event={"ID":"25b87df0-e39e-40bc-9500-09807efa8334","Type":"ContainerDied","Data":"f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419"} Nov 01 09:20:23.367760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.367603 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68" event={"ID":"25b87df0-e39e-40bc-9500-09807efa8334","Type":"ContainerDied","Data":"98534d3e9dc12018cfbd78e845ab3466ce9dbba169a16c273ec279423ac12b72"} Nov 01 09:20:23.367760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.367637 5710 scope.go:122] "RemoveContainer" containerID="f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419" Nov 01 09:20:23.378167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.378150 5710 scope.go:122] "RemoveContainer" containerID="f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419" Nov 01 09:20:23.378503 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:23.378480 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419\": container with ID starting with f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419 not found: ID does not exist" containerID="f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419" Nov 01 09:20:23.378580 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.378515 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419"} err="failed to get container status \"f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419\": rpc error: code = NotFound desc = could not find container \"f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419\": container with ID starting with f6be30b7aff4beeca87581caa9016a48a019e08540c733b24e406823f5e30419 not found: ID does not exist" Nov 01 09:20:23.548527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.548489 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68"] Nov 01 09:20:23.555078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:23.554959 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0468a5-8574958446-x9d68"] Nov 01 09:20:25.059187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.059153 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b87df0-e39e-40bc-9500-09807efa8334" path="/var/lib/kubelet/pods/25b87df0-e39e-40bc-9500-09807efa8334/volumes" Nov 01 09:20:25.604175 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.604142 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:20:25.781708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.781670 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume\") pod \"e279588d-21f3-41b5-88a5-676361e5e2a8\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " Nov 01 09:20:25.781955 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.781755 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz\" (UniqueName: \"kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz\") pod \"e279588d-21f3-41b5-88a5-676361e5e2a8\" (UID: \"e279588d-21f3-41b5-88a5-676361e5e2a8\") " Nov 01 09:20:25.782116 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.782090 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume" pod "e279588d-21f3-41b5-88a5-676361e5e2a8" (UID: "e279588d-21f3-41b5-88a5-676361e5e2a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:20:25.782424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.782406 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e279588d-21f3-41b5-88a5-676361e5e2a8-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:25.784160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.784140 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz" pod "e279588d-21f3-41b5-88a5-676361e5e2a8" (UID: "e279588d-21f3-41b5-88a5-676361e5e2a8"). InnerVolumeSpecName "kube-api-access-vdccz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:20:25.883415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:25.883377 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdccz\" (UniqueName: \"kubernetes.io/projected/e279588d-21f3-41b5-88a5-676361e5e2a8-kube-api-access-vdccz\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:26.390267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.390235 5710 generic.go:356] "Generic (PLEG): container finished" podID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerID="fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1" exitCode=0 Nov 01 09:20:26.390745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.390302 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" Nov 01 09:20:26.390745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.390316 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" event={"ID":"e279588d-21f3-41b5-88a5-676361e5e2a8","Type":"ContainerDied","Data":"fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1"} Nov 01 09:20:26.390745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.390371 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr" event={"ID":"e279588d-21f3-41b5-88a5-676361e5e2a8","Type":"ContainerDied","Data":"e3a2ea0d0866ff712a34855bb22d8e91da67eb12d08e5e7b2aa388c7ecbf63d9"} Nov 01 09:20:26.390745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.390399 5710 scope.go:122] "RemoveContainer" containerID="fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1" Nov 01 09:20:26.402751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.402728 5710 scope.go:122] "RemoveContainer" containerID="fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1" Nov 01 09:20:26.403104 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:26.403077 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1\": container with ID starting with fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1 not found: ID does not exist" containerID="fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1" Nov 01 09:20:26.403222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.403111 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1"} err="failed to get container status \"fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1\": rpc error: code = NotFound desc = could not find container \"fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1\": container with ID starting with fb13c021a093b3422eb09f694cff5092af8361e0ce82e70f81579c8f6d3fd6c1 not found: ID does not exist" Nov 01 09:20:26.547662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.547620 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr"] Nov 01 09:20:26.593650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:26.593615 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6752e4-56d78d64bf-lx5cr"] Nov 01 09:20:27.059961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:27.059929 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" path="/var/lib/kubelet/pods/e279588d-21f3-41b5-88a5-676361e5e2a8/volumes" Nov 01 09:20:30.513774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:30.513729 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq"] Nov 01 09:20:30.514204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:30.514015 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" containerID="cri-o://8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7" gracePeriod=30 Nov 01 09:20:35.554109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:35.554074 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 09:20:35.554552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:35.554146 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 09:20:35.896585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:35.896552 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:20:36.034142 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034104 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034198 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034242 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034292 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034390 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034451 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034557 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034557 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:20:36.034695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034609 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034656 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs\") pod \"930bec2c-702a-4b9f-aa04-c1afb3072c24\" (UID: \"930bec2c-702a-4b9f-aa04-c1afb3072c24\") " Nov 01 09:20:36.034695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034678 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:20:36.034896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034707 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:20:36.034896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.034823 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:20:36.035284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.035259 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.035509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.035492 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930bec2c-702a-4b9f-aa04-c1afb3072c24-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.035635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.035612 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.035794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.035638 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.035794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.035529 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:20:36.037194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.037094 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:20:36.037504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.037481 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "kube-api-access-d6smd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:20:36.037755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.037687 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:20:36.037847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.037801 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo" pod "930bec2c-702a-4b9f-aa04-c1afb3072c24" (UID: "930bec2c-702a-4b9f-aa04-c1afb3072c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:20:36.136551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.136500 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.136551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.136543 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6smd\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-kube-api-access-d6smd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.136551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.136558 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.136871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.136571 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930bec2c-702a-4b9f-aa04-c1afb3072c24-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.136871 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.136583 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930bec2c-702a-4b9f-aa04-c1afb3072c24-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:36.460284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.460185 5710 generic.go:356] "Generic (PLEG): container finished" podID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerID="8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7" exitCode=0 Nov 01 09:20:36.460284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.460267 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" Nov 01 09:20:36.460284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.460264 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" event={"ID":"930bec2c-702a-4b9f-aa04-c1afb3072c24","Type":"ContainerDied","Data":"8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7"} Nov 01 09:20:36.460575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.460321 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq" event={"ID":"930bec2c-702a-4b9f-aa04-c1afb3072c24","Type":"ContainerDied","Data":"04a50653dcf53861a6b637fc951abd32c4238bd4a02aedf12ad5e5c9cf00df53"} Nov 01 09:20:36.460575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.460343 5710 scope.go:122] "RemoveContainer" containerID="8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7" Nov 01 09:20:36.471253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.471226 5710 scope.go:122] "RemoveContainer" containerID="3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f" Nov 01 09:20:36.507958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.498419 5710 scope.go:122] "RemoveContainer" containerID="8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7" Nov 01 09:20:36.507958 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:36.498932 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7\": container with ID starting with 8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7 not found: ID does not exist" containerID="8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7" Nov 01 09:20:36.507958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.498969 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7"} err="failed to get container status \"8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7\": rpc error: code = NotFound desc = could not find container \"8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7\": container with ID starting with 8d4420ea81882881ccd0cb96132d94147072358e9f0c4db7a0d8f59778dd59f7 not found: ID does not exist" Nov 01 09:20:36.507958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.498999 5710 scope.go:122] "RemoveContainer" containerID="3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f" Nov 01 09:20:36.507958 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:36.499350 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f\": container with ID starting with 3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f not found: ID does not exist" containerID="3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f" Nov 01 09:20:36.507958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.499377 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f"} err="failed to get container status \"3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f\": rpc error: code = NotFound desc = could not find container \"3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f\": container with ID starting with 3eae5cc33bac285834f0dbb23e01fe97b09dbd6b672c041953f5ad18776ef06f not found: ID does not exist" Nov 01 09:20:36.615966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.615918 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq"] Nov 01 09:20:36.661844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:36.661810 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0468a5-openshift-default-7764b7f6b9-kx9sq"] Nov 01 09:20:37.061460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:37.061417 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" path="/var/lib/kubelet/pods/930bec2c-702a-4b9f-aa04-c1afb3072c24/volumes" Nov 01 09:20:38.121586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:38.121550 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd"] Nov 01 09:20:38.223978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:38.223944 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb"] Nov 01 09:20:43.122455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.122392 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" podUID="8cdb0386-ef38-460e-847c-0aa3389ff2db" containerName="payload-processing" containerID="cri-o://8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960" gracePeriod=25 Nov 01 09:20:43.225014 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.224904 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" podUID="cbaac034-4965-4f15-8a7a-9a572cd7a5f8" containerName="payload-pre-processing" containerID="cri-o://b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7" gracePeriod=25 Nov 01 09:20:43.449209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.448825 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:20:43.539113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.539085 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:20:43.553432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.553396 5710 generic.go:356] "Generic (PLEG): container finished" podID="cbaac034-4965-4f15-8a7a-9a572cd7a5f8" containerID="b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7" exitCode=0 Nov 01 09:20:43.553643 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.553455 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" Nov 01 09:20:43.553643 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.553454 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" event={"ID":"cbaac034-4965-4f15-8a7a-9a572cd7a5f8","Type":"ContainerDied","Data":"b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7"} Nov 01 09:20:43.553643 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.553615 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb" event={"ID":"cbaac034-4965-4f15-8a7a-9a572cd7a5f8","Type":"ContainerDied","Data":"ae22e3614826c5c0b24db35cd1fada543c7d6b6aeca8c57f38124de1c6d45d19"} Nov 01 09:20:43.553806 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.553656 5710 scope.go:122] "RemoveContainer" containerID="b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7" Nov 01 09:20:43.555641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.555613 5710 generic.go:356] "Generic (PLEG): container finished" podID="8cdb0386-ef38-460e-847c-0aa3389ff2db" containerID="8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960" exitCode=0 Nov 01 09:20:43.555820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.555646 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" event={"ID":"8cdb0386-ef38-460e-847c-0aa3389ff2db","Type":"ContainerDied","Data":"8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960"} Nov 01 09:20:43.555820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.555684 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" Nov 01 09:20:43.555820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.555737 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd" event={"ID":"8cdb0386-ef38-460e-847c-0aa3389ff2db","Type":"ContainerDied","Data":"e5566d37f01869d973bb3f429fc61cd17c10c98382965d73cc917806d8662113"} Nov 01 09:20:43.569594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.569566 5710 scope.go:122] "RemoveContainer" containerID="b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7" Nov 01 09:20:43.570043 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:43.570012 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7\": container with ID starting with b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7 not found: ID does not exist" containerID="b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7" Nov 01 09:20:43.570158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.570053 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7"} err="failed to get container status \"b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7\": rpc error: code = NotFound desc = could not find container \"b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7\": container with ID starting with b1f6544e7b43e0a17a8e574ee7dae9ab457839d1717330f49b7a98383a0094e7 not found: ID does not exist" Nov 01 09:20:43.570158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.570080 5710 scope.go:122] "RemoveContainer" containerID="8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960" Nov 01 09:20:43.576068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.576022 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4\" (UniqueName: \"kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4\") pod \"8cdb0386-ef38-460e-847c-0aa3389ff2db\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " Nov 01 09:20:43.576235 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.576149 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume\") pod \"8cdb0386-ef38-460e-847c-0aa3389ff2db\" (UID: \"8cdb0386-ef38-460e-847c-0aa3389ff2db\") " Nov 01 09:20:43.576308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.576234 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume\") pod \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " Nov 01 09:20:43.576365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.576343 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9\" (UniqueName: \"kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9\") pod \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\" (UID: \"cbaac034-4965-4f15-8a7a-9a572cd7a5f8\") " Nov 01 09:20:43.577576 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.577424 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume" pod "8cdb0386-ef38-460e-847c-0aa3389ff2db" (UID: "8cdb0386-ef38-460e-847c-0aa3389ff2db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:20:43.577833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.577806 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume" pod "cbaac034-4965-4f15-8a7a-9a572cd7a5f8" (UID: "cbaac034-4965-4f15-8a7a-9a572cd7a5f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:20:43.588629 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.588597 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4" pod "8cdb0386-ef38-460e-847c-0aa3389ff2db" (UID: "8cdb0386-ef38-460e-847c-0aa3389ff2db"). InnerVolumeSpecName "kube-api-access-h5js4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:20:43.588759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.588651 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9" pod "cbaac034-4965-4f15-8a7a-9a572cd7a5f8" (UID: "cbaac034-4965-4f15-8a7a-9a572cd7a5f8"). InnerVolumeSpecName "kube-api-access-77zx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:20:43.588759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.588694 5710 scope.go:122] "RemoveContainer" containerID="8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960" Nov 01 09:20:43.589232 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:43.589192 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960\": container with ID starting with 8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960 not found: ID does not exist" containerID="8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960" Nov 01 09:20:43.589378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.589245 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960"} err="failed to get container status \"8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960\": rpc error: code = NotFound desc = could not find container \"8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960\": container with ID starting with 8f0f49ba4b546a5669d829cc722b607ac52f4fd9833477fc80b5a1fcfba5f960 not found: ID does not exist" Nov 01 09:20:43.677688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.677597 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zx9\" (UniqueName: \"kubernetes.io/projected/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-kube-api-access-77zx9\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:43.677688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.677631 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5js4\" (UniqueName: \"kubernetes.io/projected/8cdb0386-ef38-460e-847c-0aa3389ff2db-kube-api-access-h5js4\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:43.677688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.677641 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb0386-ef38-460e-847c-0aa3389ff2db-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:43.677688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.677651 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbaac034-4965-4f15-8a7a-9a572cd7a5f8-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:20:43.970291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.970185 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb"] Nov 01 09:20:43.973259 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:43.973227 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3b7524-6b9c4c5555-p6kjb"] Nov 01 09:20:44.075662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:44.075616 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd"] Nov 01 09:20:44.079601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:44.079566 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3b7524-75964d6d4c-7zrrd"] Nov 01 09:20:45.060982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:45.060947 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cdb0386-ef38-460e-847c-0aa3389ff2db" path="/var/lib/kubelet/pods/8cdb0386-ef38-460e-847c-0aa3389ff2db/volumes" Nov 01 09:20:45.061399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:45.061385 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbaac034-4965-4f15-8a7a-9a572cd7a5f8" path="/var/lib/kubelet/pods/cbaac034-4965-4f15-8a7a-9a572cd7a5f8/volumes" Nov 01 09:20:45.479471 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:20:45.479427 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:20:46.593109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:46.593073 5710 generic.go:356] "Generic (PLEG): container finished" podID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerID="d40e5ba0cd09d18a472146ea2184ab0b7cda6415774f8412057c5284cb5e6cb3" exitCode=0 Nov 01 09:20:46.593648 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:46.593133 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" event={"ID":"ed492f1a-d299-43d0-a059-af6d93321d0e","Type":"ContainerDied","Data":"d40e5ba0cd09d18a472146ea2184ab0b7cda6415774f8412057c5284cb5e6cb3"} Nov 01 09:20:46.593648 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:46.593567 5710 scope.go:122] "RemoveContainer" containerID="d40e5ba0cd09d18a472146ea2184ab0b7cda6415774f8412057c5284cb5e6cb3" Nov 01 09:20:46.595244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:46.595215 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:20:46.595314 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:46.595289 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:20:46.595350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:46.595322 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:20:47.611792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:47.611749 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" event={"ID":"ed492f1a-d299-43d0-a059-af6d93321d0e","Type":"ContainerStarted","Data":"f5677dbfaadd1b2f55b984e4de1b257c7d902f7c33ab661d65b62ad2a35cb4d8"} Nov 01 09:20:47.848241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:47.848201 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:20:47.849698 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:47.849672 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 09:20:47.849824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:47.849743 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 09:20:48.848200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:48.848158 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 09:20:48.848661 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:48.848230 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 09:20:49.848267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:49.848231 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 09:20:49.848708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:49.848296 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 09:20:50.848614 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:50.848570 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 09:20:50.849179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:50.848655 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 09:20:51.848309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:51.848271 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 09:20:51.848512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:51.848325 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 09:20:52.370996 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:52.370956 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:20:52.848592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:52.848555 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 09:20:52.848832 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:52.848626 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 09:20:53.853263 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:53.853228 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:20:53.854488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:53.854463 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:20:57.179903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.179854 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb"] Nov 01 09:20:57.180558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180523 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cbaac034-4965-4f15-8a7a-9a572cd7a5f8" containerName="payload-pre-processing" Nov 01 09:20:57.180558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180537 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaac034-4965-4f15-8a7a-9a572cd7a5f8" containerName="payload-pre-processing" Nov 01 09:20:57.180558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180554 5710 cpu_manager.go:414] "containerMap: removing container" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" Nov 01 09:20:57.180558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180560 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180573 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8cdb0386-ef38-460e-847c-0aa3389ff2db" containerName="payload-processing" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180578 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cdb0386-ef38-460e-847c-0aa3389ff2db" containerName="payload-processing" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180600 5710 cpu_manager.go:414] "containerMap: removing container" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180606 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180617 5710 cpu_manager.go:414] "containerMap: removing container" podUID="25b87df0-e39e-40bc-9500-09807efa8334" containerName="maas-api" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180621 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b87df0-e39e-40bc-9500-09807efa8334" containerName="maas-api" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180638 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerName="payload-processing" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180643 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerName="payload-processing" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180762 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cdb0386-ef38-460e-847c-0aa3389ff2db" containerName="payload-processing" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180774 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180783 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b87df0-e39e-40bc-9500-09807efa8334" containerName="maas-api" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180791 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e279588d-21f3-41b5-88a5-676361e5e2a8" containerName="payload-processing" Nov 01 09:20:57.180877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.180799 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaac034-4965-4f15-8a7a-9a572cd7a5f8" containerName="payload-pre-processing" Nov 01 09:20:57.184567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.184544 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.347121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.342936 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.361986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.361952 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.362429 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.362403 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.363426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.363102 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.364407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.364378 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.366626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.366600 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.367022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.367000 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.367985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.367960 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.368143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.368127 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mfh\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.368423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.349490 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-65aeef-openshift-default-dockercfg-5fxbb\"" Nov 01 09:20:57.369859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.369817 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb"] Nov 01 09:20:57.469640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469551 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.469640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469598 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.469640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469616 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mfh\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.469961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469693 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.469961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469773 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.469961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469816 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.469961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469933 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.469974 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.470053 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.470064 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.470117 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470186 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.470153 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.470427 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.470585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.470565 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.472532 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.472507 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.472822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.472801 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.527570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.527524 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mfh\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.527788 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.527585 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token\") pod \"e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:57.799864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:57.799767 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:58.143732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.143685 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:20:58.288593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.288557 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb"] Nov 01 09:20:58.295813 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:20:58.295781 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9478fa58_762a_45cc_aa2f_91803a661f30.slice/crio-380efa718ef6138595678fcca1ff0919a4e1c3e9ebc4ad3523b97fd3dd060dc5 WatchSource:0}: Error finding container 380efa718ef6138595678fcca1ff0919a4e1c3e9ebc4ad3523b97fd3dd060dc5: Status 404 returned error can't find the container with id 380efa718ef6138595678fcca1ff0919a4e1c3e9ebc4ad3523b97fd3dd060dc5 Nov 01 09:20:58.298617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.298584 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:20:58.298704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.298655 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:20:58.298704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.298684 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:20:58.778372 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.778323 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" event={"ID":"9478fa58-762a-45cc-aa2f-91803a661f30","Type":"ContainerStarted","Data":"4cd84c1306ce8ebcdc041f515d4bd736e2874e5d2877a689375823ac6d588db1"} Nov 01 09:20:58.778584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.778396 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" event={"ID":"9478fa58-762a-45cc-aa2f-91803a661f30","Type":"ContainerStarted","Data":"380efa718ef6138595678fcca1ff0919a4e1c3e9ebc4ad3523b97fd3dd060dc5"} Nov 01 09:20:58.799905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.799862 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:20:58.905485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:58.905394 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podStartSLOduration=1.9053734919999998 podStartE2EDuration="1.905373492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:20:58.902815308 +0000 UTC m=+2881.991151980" watchObservedRunningTime="2026-11-01 09:20:58.905373492 +0000 UTC m=+2881.993710166" Nov 01 09:20:59.806617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:20:59.806577 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:21:00.791502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:00.791470 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:21:00.792760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:00.792705 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:21:02.118746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.101999 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh"] Nov 01 09:21:02.130031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.129977 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="930bec2c-702a-4b9f-aa04-c1afb3072c24" containerName="istio-proxy" Nov 01 09:21:02.138201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.138168 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh"] Nov 01 09:21:02.138368 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.138310 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.187342 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.187306 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5b96ac\"" Nov 01 09:21:02.233635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.233597 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5b96ac-dockercfg-7xhlz\"" Nov 01 09:21:02.293651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.293602 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtsq\" (UniqueName: \"kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq\") pod \"payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.293887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.293672 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.395043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.394952 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtsq\" (UniqueName: \"kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq\") pod \"payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.395043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.395030 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.395793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.395772 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.453801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.453769 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtsq\" (UniqueName: \"kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq\") pod \"payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.459563 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.459536 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:02.840660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.840629 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh"] Nov 01 09:21:02.847401 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:21:02.847372 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db332be_99b7_4989_bd7e_c2c07a30ee53.slice/crio-063182b91a965b735e8de0242155ca271af95897234139b446efb1b731e6ea16 WatchSource:0}: Error finding container 063182b91a965b735e8de0242155ca271af95897234139b446efb1b731e6ea16: Status 404 returned error can't find the container with id 063182b91a965b735e8de0242155ca271af95897234139b446efb1b731e6ea16 Nov 01 09:21:02.865953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:02.865919 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" event={"ID":"0db332be-99b7-4989-bd7e-c2c07a30ee53","Type":"ContainerStarted","Data":"063182b91a965b735e8de0242155ca271af95897234139b446efb1b731e6ea16"} Nov 01 09:21:03.874185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:03.874149 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" event={"ID":"0db332be-99b7-4989-bd7e-c2c07a30ee53","Type":"ContainerStarted","Data":"f1960fa6850b8ac2432e47f28f28d94632f276fa8c79817a1bce70314369db36"} Nov 01 09:21:03.874693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:03.874215 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:03.989129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:03.989071 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" podStartSLOduration=1.989053908 podStartE2EDuration="1.989053908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:03.987621528 +0000 UTC m=+2887.075958201" watchObservedRunningTime="2026-11-01 09:21:03.989053908 +0000 UTC m=+2887.077390580" Nov 01 09:21:06.764023 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.763991 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4"] Nov 01 09:21:06.774941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.774899 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:06.812751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.812401 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4"] Nov 01 09:21:06.852733 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.852686 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-65aeef-dockercfg-7clwg\"" Nov 01 09:21:06.852923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.852763 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-65aeef\"" Nov 01 09:21:06.868524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.868484 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:06.868786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.868757 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9lq4\" (UniqueName: \"kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4\") pod \"payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:06.969824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.969784 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9lq4\" (UniqueName: \"kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4\") pod \"payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:06.969989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.969898 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:06.970645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:06.970623 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:07.027749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:07.027632 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9lq4\" (UniqueName: \"kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4\") pod \"payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:07.112669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:07.112621 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:07.497865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:07.497821 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4"] Nov 01 09:21:07.504400 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:21:07.504358 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9489fe56_209e_4fef_b29b_85e9b338ccf5.slice/crio-fb236a58e72e3879fe2617e4433f0c6a00f9d8085e745f2a101e59fbceed95e5 WatchSource:0}: Error finding container fb236a58e72e3879fe2617e4433f0c6a00f9d8085e745f2a101e59fbceed95e5: Status 404 returned error can't find the container with id fb236a58e72e3879fe2617e4433f0c6a00f9d8085e745f2a101e59fbceed95e5 Nov 01 09:21:07.901283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:07.901231 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" event={"ID":"9489fe56-209e-4fef-b29b-85e9b338ccf5","Type":"ContainerStarted","Data":"44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff"} Nov 01 09:21:07.901776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:07.901304 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" event={"ID":"9489fe56-209e-4fef-b29b-85e9b338ccf5","Type":"ContainerStarted","Data":"fb236a58e72e3879fe2617e4433f0c6a00f9d8085e745f2a101e59fbceed95e5"} Nov 01 09:21:07.901776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:07.901331 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:08.014594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:08.014537 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" podStartSLOduration=2.014521645 podStartE2EDuration="2.014521645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:08.011986395 +0000 UTC m=+2891.100323078" watchObservedRunningTime="2026-11-01 09:21:08.014521645 +0000 UTC m=+2891.102858317" Nov 01 09:21:14.884435 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:14.884395 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:21:15.479366 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:15.479314 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:21:18.910628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:18.910594 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 09:21:26.147839 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.147803 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq"] Nov 01 09:21:26.152121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.152097 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.174645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.174606 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.174842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.174781 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwsv\" (UniqueName: \"kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.268389 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.268336 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-46bf99\"" Nov 01 09:21:26.268705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.268367 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-46bf99-dockercfg-7bcnn\"" Nov 01 09:21:26.285753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.279462 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.287017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.286964 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwsv\" (UniqueName: \"kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.298744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.288855 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.350508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.350432 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq"] Nov 01 09:21:26.350979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.350955 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp"] Nov 01 09:21:26.368742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.356852 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.390177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.389887 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhgp\" (UniqueName: \"kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp\") pod \"payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.390177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.390039 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.391282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.391244 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp"] Nov 01 09:21:26.393495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.393459 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwsv\" (UniqueName: \"kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.467276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.467233 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:26.492779 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.491544 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.492779 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.491809 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhgp\" (UniqueName: \"kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp\") pod \"payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.492779 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.492710 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.551261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.551166 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhgp\" (UniqueName: \"kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp\") pod \"payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.717111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.717014 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:26.894771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:26.894664 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq"] Nov 01 09:21:26.901958 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:21:26.901923 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4a42e2_3ca9_4fd5_85cc_97760d63ca49.slice/crio-9d00b9f181579e385db1d4a81cea749e08342fd40a0b6afe9e07be902b98beec WatchSource:0}: Error finding container 9d00b9f181579e385db1d4a81cea749e08342fd40a0b6afe9e07be902b98beec: Status 404 returned error can't find the container with id 9d00b9f181579e385db1d4a81cea749e08342fd40a0b6afe9e07be902b98beec Nov 01 09:21:27.098609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:27.098568 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp"] Nov 01 09:21:27.106061 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:21:27.106031 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635db112_2d4d_436d_9ceb_9a7e80314b4e.slice/crio-73afd9d3b8409f3fc91c0b630d31e131e07a96169cc6bd906a5ccb406091fb56 WatchSource:0}: Error finding container 73afd9d3b8409f3fc91c0b630d31e131e07a96169cc6bd906a5ccb406091fb56: Status 404 returned error can't find the container with id 73afd9d3b8409f3fc91c0b630d31e131e07a96169cc6bd906a5ccb406091fb56 Nov 01 09:21:27.112001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:27.111978 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" event={"ID":"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49","Type":"ContainerStarted","Data":"076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180"} Nov 01 09:21:27.112089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:27.112038 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" event={"ID":"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49","Type":"ContainerStarted","Data":"9d00b9f181579e385db1d4a81cea749e08342fd40a0b6afe9e07be902b98beec"} Nov 01 09:21:27.112140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:27.112083 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:27.222479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:27.222360 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" podStartSLOduration=1.22234365 podStartE2EDuration="1.22234365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:27.219404744 +0000 UTC m=+2910.307741416" watchObservedRunningTime="2026-11-01 09:21:27.22234365 +0000 UTC m=+2910.310680321" Nov 01 09:21:28.120376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:28.120334 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" event={"ID":"635db112-2d4d-436d-9ceb-9a7e80314b4e","Type":"ContainerStarted","Data":"04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b"} Nov 01 09:21:28.120599 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:28.120399 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" event={"ID":"635db112-2d4d-436d-9ceb-9a7e80314b4e","Type":"ContainerStarted","Data":"73afd9d3b8409f3fc91c0b630d31e131e07a96169cc6bd906a5ccb406091fb56"} Nov 01 09:21:28.120649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:28.120614 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:28.233633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:28.233575 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" podStartSLOduration=2.23355783 podStartE2EDuration="2.23355783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:21:28.230182254 +0000 UTC m=+2911.318518925" watchObservedRunningTime="2026-11-01 09:21:28.23355783 +0000 UTC m=+2911.321894503" Nov 01 09:21:31.132914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.132875 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm"] Nov 01 09:21:31.140497 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.140442 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" podUID="a5419681-84fc-4716-ab19-16908093c3a4" containerName="maas-api" containerID="cri-o://af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913" gracePeriod=30 Nov 01 09:21:31.501635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.501604 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:21:31.591745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.591577 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd\" (UniqueName: \"kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd\") pod \"a5419681-84fc-4716-ab19-16908093c3a4\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " Nov 01 09:21:31.591745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.591651 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs\") pod \"a5419681-84fc-4716-ab19-16908093c3a4\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " Nov 01 09:21:31.595743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.592052 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca\") pod \"a5419681-84fc-4716-ab19-16908093c3a4\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " Nov 01 09:21:31.595743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.592136 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\") pod \"a5419681-84fc-4716-ab19-16908093c3a4\" (UID: \"a5419681-84fc-4716-ab19-16908093c3a4\") " Nov 01 09:21:31.595743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.595357 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca" pod "a5419681-84fc-4716-ab19-16908093c3a4" (UID: "a5419681-84fc-4716-ab19-16908093c3a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:21:31.597207 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.597138 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls" pod "a5419681-84fc-4716-ab19-16908093c3a4" (UID: "a5419681-84fc-4716-ab19-16908093c3a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:21:31.603987 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.603953 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd" pod "a5419681-84fc-4716-ab19-16908093c3a4" (UID: "a5419681-84fc-4716-ab19-16908093c3a4"). InnerVolumeSpecName "kube-api-access-vxlkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:21:31.604633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.604587 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs" pod "a5419681-84fc-4716-ab19-16908093c3a4" (UID: "a5419681-84fc-4716-ab19-16908093c3a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:21:31.694524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.694483 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5419681-84fc-4716-ab19-16908093c3a4-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:31.694524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.694517 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:31.694524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.694528 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxlkd\" (UniqueName: \"kubernetes.io/projected/a5419681-84fc-4716-ab19-16908093c3a4-kube-api-access-vxlkd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:31.694524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:31.694537 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5419681-84fc-4716-ab19-16908093c3a4-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:32.218903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.218859 5710 generic.go:356] "Generic (PLEG): container finished" podID="a5419681-84fc-4716-ab19-16908093c3a4" containerID="af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913" exitCode=0 Nov 01 09:21:32.222946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.218934 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" event={"ID":"a5419681-84fc-4716-ab19-16908093c3a4","Type":"ContainerDied","Data":"af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913"} Nov 01 09:21:32.222946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.218993 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" event={"ID":"a5419681-84fc-4716-ab19-16908093c3a4","Type":"ContainerDied","Data":"9017a6f20ecd384febd07d3d7a520e1de8c1754471b958080a704c487de7f6b9"} Nov 01 09:21:32.222946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.219025 5710 scope.go:122] "RemoveContainer" containerID="af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913" Nov 01 09:21:32.222946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.219246 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm" Nov 01 09:21:32.250510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.250151 5710 scope.go:122] "RemoveContainer" containerID="af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913" Nov 01 09:21:32.250624 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:32.250546 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913\": container with ID starting with af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913 not found: ID does not exist" containerID="af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913" Nov 01 09:21:32.250624 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.250581 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913"} err="failed to get container status \"af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913\": rpc error: code = NotFound desc = could not find container \"af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913\": container with ID starting with af216276357d59db0728c10b5cf13e4e44d3e8e9026ba42440e459a663bc3913 not found: ID does not exist" Nov 01 09:21:32.377528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.377488 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm"] Nov 01 09:21:32.423701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:32.423662 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3b7524-79fc7f4c6b-vvsdm"] Nov 01 09:21:33.059566 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:33.059527 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5419681-84fc-4716-ab19-16908093c3a4" path="/var/lib/kubelet/pods/a5419681-84fc-4716-ab19-16908093c3a4/volumes" Nov 01 09:21:38.123892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:38.123859 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:21:39.130475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:39.130436 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:21:43.250742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.250693 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5"] Nov 01 09:21:43.251279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.250930 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" podUID="57ab4c3c-3c9b-4844-a6ab-02fb508c750b" containerName="maas-api" containerID="cri-o://d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40" gracePeriod=30 Nov 01 09:21:43.602894 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.602862 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:21:43.704750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.704348 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca\") pod \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " Nov 01 09:21:43.704750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.704464 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln\" (UniqueName: \"kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln\") pod \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " Nov 01 09:21:43.704750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.704527 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\") pod \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " Nov 01 09:21:43.704750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.704661 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs\") pod \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\" (UID: \"57ab4c3c-3c9b-4844-a6ab-02fb508c750b\") " Nov 01 09:21:43.708209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.708149 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca" pod "57ab4c3c-3c9b-4844-a6ab-02fb508c750b" (UID: "57ab4c3c-3c9b-4844-a6ab-02fb508c750b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:21:43.713283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.713217 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs" pod "57ab4c3c-3c9b-4844-a6ab-02fb508c750b" (UID: "57ab4c3c-3c9b-4844-a6ab-02fb508c750b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:21:43.729362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.729306 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln" pod "57ab4c3c-3c9b-4844-a6ab-02fb508c750b" (UID: "57ab4c3c-3c9b-4844-a6ab-02fb508c750b"). InnerVolumeSpecName "kube-api-access-gcxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:21:43.729878 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.729700 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls" pod "57ab4c3c-3c9b-4844-a6ab-02fb508c750b" (UID: "57ab4c3c-3c9b-4844-a6ab-02fb508c750b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:21:43.809427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.809386 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:43.809427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.809423 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:43.809427 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.809433 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxln\" (UniqueName: \"kubernetes.io/projected/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-kube-api-access-gcxln\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:43.809697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:43.809445 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57ab4c3c-3c9b-4844-a6ab-02fb508c750b-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:44.337741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.333194 5710 generic.go:356] "Generic (PLEG): container finished" podID="57ab4c3c-3c9b-4844-a6ab-02fb508c750b" containerID="d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40" exitCode=0 Nov 01 09:21:44.337741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.333249 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" event={"ID":"57ab4c3c-3c9b-4844-a6ab-02fb508c750b","Type":"ContainerDied","Data":"d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40"} Nov 01 09:21:44.337741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.333305 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" event={"ID":"57ab4c3c-3c9b-4844-a6ab-02fb508c750b","Type":"ContainerDied","Data":"8963e14249ffb6ac8c205d38b76096b608b970597ba1ccfe1b1f466550feef8d"} Nov 01 09:21:44.337741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.333336 5710 scope.go:122] "RemoveContainer" containerID="d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40" Nov 01 09:21:44.337741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.333512 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5" Nov 01 09:21:44.361068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.361036 5710 scope.go:122] "RemoveContainer" containerID="d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40" Nov 01 09:21:44.361536 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:44.361497 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40\": container with ID starting with d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40 not found: ID does not exist" containerID="d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40" Nov 01 09:21:44.361679 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.361548 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40"} err="failed to get container status \"d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40\": rpc error: code = NotFound desc = could not find container \"d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40\": container with ID starting with d115774903465bf6d932b0450deb7fa8630149fba38eb86223be1f1fcb9f7d40 not found: ID does not exist" Nov 01 09:21:44.502009 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.501976 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5"] Nov 01 09:21:44.547945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:44.547910 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6752e4-8c74b6895-77vb5"] Nov 01 09:21:45.059788 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:45.059744 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ab4c3c-3c9b-4844-a6ab-02fb508c750b" path="/var/lib/kubelet/pods/57ab4c3c-3c9b-4844-a6ab-02fb508c750b/volumes" Nov 01 09:21:45.479476 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:45.479436 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:21:45.833956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:45.833861 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq"] Nov 01 09:21:45.834148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:45.834087 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" containerID="cri-o://6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56" gracePeriod=30 Nov 01 09:21:48.883472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:48.883428 5710 patch_prober.go:29] interesting pod/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:21:48.883937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:48.883508 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:21:51.122028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.121955 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:21:51.222631 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222593 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.222934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222665 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.222934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222744 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.222934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222776 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.222934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222897 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.222934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222930 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.223253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222964 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:21:51.223253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222978 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.223253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.222979 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:21:51.223253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.223104 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.223253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.223226 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss\") pod \"200c3913-22d9-4f43-93cd-03ac77c8f7de\" (UID: \"200c3913-22d9-4f43-93cd-03ac77c8f7de\") " Nov 01 09:21:51.223527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.223303 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:21:51.223527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.223420 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:21:51.223642 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.223548 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:21:51.224652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.224346 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.224652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.224532 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.224652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.224565 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.224652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.224580 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.224652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.224594 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200c3913-22d9-4f43-93cd-03ac77c8f7de-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.226248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.226207 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:21:51.226404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.226243 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:21:51.226477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.226452 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:21:51.226678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.226650 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss" pod "200c3913-22d9-4f43-93cd-03ac77c8f7de" (UID: "200c3913-22d9-4f43-93cd-03ac77c8f7de"). InnerVolumeSpecName "kube-api-access-kc7ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:21:51.326307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.326267 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.326307 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.326308 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.326531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.326323 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc7ss\" (UniqueName: \"kubernetes.io/projected/200c3913-22d9-4f43-93cd-03ac77c8f7de-kube-api-access-kc7ss\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.326531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.326340 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200c3913-22d9-4f43-93cd-03ac77c8f7de-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:21:51.406108 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.406012 5710 generic.go:356] "Generic (PLEG): container finished" podID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerID="6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56" exitCode=0 Nov 01 09:21:51.406108 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.406067 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" Nov 01 09:21:51.406108 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.406078 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" event={"ID":"200c3913-22d9-4f43-93cd-03ac77c8f7de","Type":"ContainerDied","Data":"6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56"} Nov 01 09:21:51.406415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.406138 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq" event={"ID":"200c3913-22d9-4f43-93cd-03ac77c8f7de","Type":"ContainerDied","Data":"8ea7139f778418dcf3f64b48eeecadfb202ecaf1fadeaa78f94c6d46f1cb80fe"} Nov 01 09:21:51.406415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.406160 5710 scope.go:122] "RemoveContainer" containerID="6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56" Nov 01 09:21:51.417905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.417881 5710 scope.go:122] "RemoveContainer" containerID="6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56" Nov 01 09:21:51.418252 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:51.418233 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56\": container with ID starting with 6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56 not found: ID does not exist" containerID="6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56" Nov 01 09:21:51.418320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.418266 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56"} err="failed to get container status \"6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56\": rpc error: code = NotFound desc = could not find container \"6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56\": container with ID starting with 6a899f2cccdfbeb2f722115cb8a01c5deefe73ef54da8281fd3c45d670dadb56 not found: ID does not exist" Nov 01 09:21:51.561872 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.561837 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq"] Nov 01 09:21:51.565836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:51.565809 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6752e4-openshift-default-5d799fcf57-mzvmq"] Nov 01 09:21:53.059144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:53.059103 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" path="/var/lib/kubelet/pods/200c3913-22d9-4f43-93cd-03ac77c8f7de/volumes" Nov 01 09:21:55.152792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.152752 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc"] Nov 01 09:21:55.154058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154020 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a5419681-84fc-4716-ab19-16908093c3a4" containerName="maas-api" Nov 01 09:21:55.154058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154050 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5419681-84fc-4716-ab19-16908093c3a4" containerName="maas-api" Nov 01 09:21:55.154276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154068 5710 cpu_manager.go:414] "containerMap: removing container" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" Nov 01 09:21:55.154276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154075 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" Nov 01 09:21:55.154276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154147 5710 cpu_manager.go:414] "containerMap: removing container" podUID="57ab4c3c-3c9b-4844-a6ab-02fb508c750b" containerName="maas-api" Nov 01 09:21:55.154276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154155 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ab4c3c-3c9b-4844-a6ab-02fb508c750b" containerName="maas-api" Nov 01 09:21:55.154478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154309 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ab4c3c-3c9b-4844-a6ab-02fb508c750b" containerName="maas-api" Nov 01 09:21:55.154478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154325 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5419681-84fc-4716-ab19-16908093c3a4" containerName="maas-api" Nov 01 09:21:55.154478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.154337 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="200c3913-22d9-4f43-93cd-03ac77c8f7de" containerName="istio-proxy" Nov 01 09:21:55.159024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.158996 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.208743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.208572 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.208743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.208646 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.209150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.208803 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.209150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.209036 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxxt\" (UniqueName: \"kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.209277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.209219 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5b96ac-serving-cert\"" Nov 01 09:21:55.262379 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.262342 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc"] Nov 01 09:21:55.310340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.310297 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.310340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.310339 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.310636 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:55.310437 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-serving-cert: secret "maas-api-e2e-shared-b-w3-5b96ac-serving-cert" not found Nov 01 09:21:55.310636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.310458 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.310636 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:55.310488 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls podName:242da1e1-32b9-401a-9f7d-f42b00b17609 nodeName:}" failed. No retries permitted until 2026-11-01 09:21:55.810473201 +0000 UTC m=+2938.898809852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" (UID: "242da1e1-32b9-401a-9f7d-f42b00b17609") : secret "maas-api-e2e-shared-b-w3-5b96ac-serving-cert" not found Nov 01 09:21:55.310850 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.310649 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxxt\" (UniqueName: \"kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.311090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.311068 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.313142 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.313116 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.376558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.376526 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxxt\" (UniqueName: \"kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.818428 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.818382 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:55.821264 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:55.821236 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:56.099964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:56.099922 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:58.702116 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:58.702077 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:21:58.702116 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd" Netns:"/var/run/netns/bcad8260-1400-4cca-ab1e-9668b184775b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:21:58.702116 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:21:58.702116 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:58.702151 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd" Netns:"/var/run/netns/bcad8260-1400-4cca-ab1e-9668b184775b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:58.702176 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd" Netns:"/var/run/netns/bcad8260-1400-4cca-ab1e-9668b184775b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:58.702643 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:21:58.702276 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra(242da1e1-32b9-401a-9f7d-f42b00b17609)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra(242da1e1-32b9-401a-9f7d-f42b00b17609)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd\\\" Netns:\\\"/var/run/netns/bcad8260-1400-4cca-ab1e-9668b184775b\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=069468584cdafb590541e81417fcad01ad16125515b6b50d87103a565eac0dfd;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" Nov 01 09:21:59.463776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:59.463734 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:21:59.464163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:21:59.464144 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:00.520763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.520660 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4"] Nov 01 09:22:00.526372 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.526342 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.582679 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.582641 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.582928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.582710 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqs2\" (UniqueName: \"kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.583001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.582972 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.583068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.583051 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.623298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.623258 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-65aeef-serving-cert\"" Nov 01 09:22:00.626143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.626114 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4"] Nov 01 09:22:00.684557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.684523 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqs2\" (UniqueName: \"kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.684808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.684759 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.684903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.684842 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.684997 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.684903 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.685109 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:00.685088 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-serving-cert: secret "maas-api-e2e-shared-b-w1-65aeef-serving-cert" not found Nov 01 09:22:00.685183 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:00.685169 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls podName:380d0df9-c4c5-4912-8a44-1cb19a3d8872 nodeName:}" failed. No retries permitted until 2026-11-01 09:22:01.18514802 +0000 UTC m=+2944.273484684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls") pod "maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" (UID: "380d0df9-c4c5-4912-8a44-1cb19a3d8872") : secret "maas-api-e2e-shared-b-w1-65aeef-serving-cert" not found Nov 01 09:22:00.685637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.685581 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.688125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.688100 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:00.746736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:00.746677 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqs2\" (UniqueName: \"kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:01.211506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:01.211458 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:01.214261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:01.214234 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:01.443786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:01.443748 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:02.067791 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:02.067757 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:22:02.067791 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c" Netns:"/var/run/netns/38a61991-4c71-45dc-809c-fa2c605296cb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:22:02.067791 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:02.067791 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:02.067810 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c" Netns:"/var/run/netns/38a61991-4c71-45dc-809c-fa2c605296cb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:02.067837 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c" Netns:"/var/run/netns/38a61991-4c71-45dc-809c-fa2c605296cb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:02.068247 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:02.067914 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra(242da1e1-32b9-401a-9f7d-f42b00b17609)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra(242da1e1-32b9-401a-9f7d-f42b00b17609)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c\\\" Netns:\\\"/var/run/netns/38a61991-4c71-45dc-809c-fa2c605296cb\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=b87a30ea0675bd5d9c2bdae780b81ec2a56610713cd2206d65f0dff1fe02716c;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" Nov 01 09:22:04.043487 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:04.043443 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:22:04.043487 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9" Netns:"/var/run/netns/659b1bcb-b9a2-4c6e-833f-76d6a89ccdfb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:04.043487 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:04.043487 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:04.043523 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9" Netns:"/var/run/netns/659b1bcb-b9a2-4c6e-833f-76d6a89ccdfb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:04.043551 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9" Netns:"/var/run/netns/659b1bcb-b9a2-4c6e-833f-76d6a89ccdfb" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:04.043983 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:04.043651 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra(380d0df9-c4c5-4912-8a44-1cb19a3d8872)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra(380d0df9-c4c5-4912-8a44-1cb19a3d8872)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9\\\" Netns:\\\"/var/run/netns/659b1bcb-b9a2-4c6e-833f-76d6a89ccdfb\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=db31a31cb808ea6c13816f10eb650f7d1c26c4acaa4a3b9898af624e39f0abf9;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" Nov 01 09:22:04.494692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:04.494658 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:04.495055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:04.495041 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:04.959965 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:04.959913 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8"] Nov 01 09:22:04.960254 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:04.960210 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" podUID="44712043-211d-44d7-81a6-00e4b52b24e0" containerName="maas-api" containerID="cri-o://66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688" gracePeriod=30 Nov 01 09:22:05.284313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.284286 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:22:05.391923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.391878 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs\") pod \"44712043-211d-44d7-81a6-00e4b52b24e0\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " Nov 01 09:22:05.392770 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.392745 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca\") pod \"44712043-211d-44d7-81a6-00e4b52b24e0\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " Nov 01 09:22:05.393383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.393354 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\") pod \"44712043-211d-44d7-81a6-00e4b52b24e0\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " Nov 01 09:22:05.393977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.393957 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg\" (UniqueName: \"kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg\") pod \"44712043-211d-44d7-81a6-00e4b52b24e0\" (UID: \"44712043-211d-44d7-81a6-00e4b52b24e0\") " Nov 01 09:22:05.395438 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.393283 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca" pod "44712043-211d-44d7-81a6-00e4b52b24e0" (UID: "44712043-211d-44d7-81a6-00e4b52b24e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:22:05.415551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.413706 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs" pod "44712043-211d-44d7-81a6-00e4b52b24e0" (UID: "44712043-211d-44d7-81a6-00e4b52b24e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:22:05.431744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.428167 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg" pod "44712043-211d-44d7-81a6-00e4b52b24e0" (UID: "44712043-211d-44d7-81a6-00e4b52b24e0"). InnerVolumeSpecName "kube-api-access-6zxqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:22:05.431744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.428382 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls" pod "44712043-211d-44d7-81a6-00e4b52b24e0" (UID: "44712043-211d-44d7-81a6-00e4b52b24e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:22:05.496492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.496454 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxqg\" (UniqueName: \"kubernetes.io/projected/44712043-211d-44d7-81a6-00e4b52b24e0-kube-api-access-6zxqg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:22:05.496492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.496493 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:22:05.496765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.496510 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44712043-211d-44d7-81a6-00e4b52b24e0-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:22:05.496765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.496526 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44712043-211d-44d7-81a6-00e4b52b24e0-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:22:05.502763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.502707 5710 generic.go:356] "Generic (PLEG): container finished" podID="44712043-211d-44d7-81a6-00e4b52b24e0" containerID="66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688" exitCode=0 Nov 01 09:22:05.502915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.502753 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" event={"ID":"44712043-211d-44d7-81a6-00e4b52b24e0","Type":"ContainerDied","Data":"66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688"} Nov 01 09:22:05.502915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.502785 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" Nov 01 09:22:05.502915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.502818 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8" event={"ID":"44712043-211d-44d7-81a6-00e4b52b24e0","Type":"ContainerDied","Data":"f12a3b3584d649dd7559889615de6700567fdce7336e226236d8dbbed45d2eeb"} Nov 01 09:22:05.502915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.502844 5710 scope.go:122] "RemoveContainer" containerID="66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688" Nov 01 09:22:05.514653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.514630 5710 scope.go:122] "RemoveContainer" containerID="66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688" Nov 01 09:22:05.514969 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:05.514949 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688\": container with ID starting with 66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688 not found: ID does not exist" containerID="66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688" Nov 01 09:22:05.515054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.514976 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688"} err="failed to get container status \"66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688\": rpc error: code = NotFound desc = could not find container \"66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688\": container with ID starting with 66131f1993cea961bf22207887d530e47eb4ab9a350a9c5d9b87866231652688 not found: ID does not exist" Nov 01 09:22:05.663887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.663849 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8"] Nov 01 09:22:05.669010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:05.668971 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514eb3-f59b6c899-sgmr8"] Nov 01 09:22:07.059226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:07.059182 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44712043-211d-44d7-81a6-00e4b52b24e0" path="/var/lib/kubelet/pods/44712043-211d-44d7-81a6-00e4b52b24e0/volumes" Nov 01 09:22:07.095177 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:07.095147 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:22:07.095177 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26" Netns:"/var/run/netns/14375984-4f7f-49c7-83df-2b53e4090c90" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:07.095177 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:07.095177 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:07.095198 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26" Netns:"/var/run/netns/14375984-4f7f-49c7-83df-2b53e4090c90" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:07.095226 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26" Netns:"/var/run/netns/14375984-4f7f-49c7-83df-2b53e4090c90" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:07.095362 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:07.095292 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra(380d0df9-c4c5-4912-8a44-1cb19a3d8872)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra(380d0df9-c4c5-4912-8a44-1cb19a3d8872)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26\\\" Netns:\\\"/var/run/netns/14375984-4f7f-49c7-83df-2b53e4090c90\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=8792d58260d3d29cdd66a5d0b79c1b6233992de33c134fe776847dd88f630f26;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" Nov 01 09:22:15.050626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:15.050590 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:15.051217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:15.051187 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:15.479413 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:15.479370 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:22:17.653862 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:17.653829 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:22:17.653862 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1" Netns:"/var/run/netns/329bfaac-394e-4c63-b39e-3b814d710976" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:22:17.653862 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:17.653862 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:17.653879 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1" Netns:"/var/run/netns/329bfaac-394e-4c63-b39e-3b814d710976" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:17.653899 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1" Netns:"/var/run/netns/329bfaac-394e-4c63-b39e-3b814d710976" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s": context deadline exceeded Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:17.654453 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:17.653976 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra(242da1e1-32b9-401a-9f7d-f42b00b17609)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra(242da1e1-32b9-401a-9f7d-f42b00b17609)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc_odh-ai-gateway-infra_242da1e1-32b9-401a-9f7d-f42b00b17609_0(aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1\\\" Netns:\\\"/var/run/netns/329bfaac-394e-4c63-b39e-3b814d710976\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc;K8S_POD_INFRA_CONTAINER_ID=aaead58944f32f0998574cd20d4159f43192a2ef79f9c6c7ee5b12363471eeb1;K8S_POD_UID=242da1e1-32b9-401a-9f7d-f42b00b17609\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc/242da1e1-32b9-401a-9f7d-f42b00b17609]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" Nov 01 09:22:21.070589 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.070549 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:21.071860 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.071834 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:21.743690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.743645 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq"] Nov 01 09:22:21.744522 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.744497 5710 cpu_manager.go:414] "containerMap: removing container" podUID="44712043-211d-44d7-81a6-00e4b52b24e0" containerName="maas-api" Nov 01 09:22:21.744522 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.744521 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44712043-211d-44d7-81a6-00e4b52b24e0" containerName="maas-api" Nov 01 09:22:21.744673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.744634 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="44712043-211d-44d7-81a6-00e4b52b24e0" containerName="maas-api" Nov 01 09:22:21.750279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.750256 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.770487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.770452 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:22:21.799316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.799281 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-46bf99-serving-cert\"" Nov 01 09:22:21.803113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.803083 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq"] Nov 01 09:22:21.862627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.862585 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.862959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.862645 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4m2p\" (UniqueName: \"kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.863051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.862986 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.863180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.863155 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.964434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.964396 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.964634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.964489 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.964634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.964527 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.964634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.964549 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4m2p\" (UniqueName: \"kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.964791 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:21.964682 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-serving-cert: secret "maas-api-e2e-shared-b-w4-46bf99-serving-cert" not found Nov 01 09:22:21.964831 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:21.964808 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls podName:f8adc733-8842-4850-a899-a95975c783e5 nodeName:}" failed. No retries permitted until 2026-11-01 09:22:22.464780793 +0000 UTC m=+2965.553117464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls") pod "maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" (UID: "f8adc733-8842-4850-a899-a95975c783e5") : secret "maas-api-e2e-shared-b-w4-46bf99-serving-cert" not found Nov 01 09:22:21.965154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.965132 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:21.967383 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:21.967352 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:22.077573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:22.077486 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4m2p\" (UniqueName: \"kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:22.479370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:22.479330 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:22.482790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:22.482757 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:22.669213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:22.669129 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:23.715630 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829" Netns:"/var/run/netns/a6bdc015-722a-456f-8353-cfd55d0ebd06" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:23.715705 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829" Netns:"/var/run/netns/a6bdc015-722a-456f-8353-cfd55d0ebd06" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:23.715785 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829" Netns:"/var/run/netns/a6bdc015-722a-456f-8353-cfd55d0ebd06" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s": context deadline exceeded Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:23.715860 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:23.716895 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:23.715896 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra(380d0df9-c4c5-4912-8a44-1cb19a3d8872)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra(380d0df9-c4c5-4912-8a44-1cb19a3d8872)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4_odh-ai-gateway-infra_380d0df9-c4c5-4912-8a44-1cb19a3d8872_0(a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829\\\" Netns:\\\"/var/run/netns/a6bdc015-722a-456f-8353-cfd55d0ebd06\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4;K8S_POD_INFRA_CONTAINER_ID=a83eb7e88adb5b3dffd1bcbd1393b3b9e1d66b57c9546e5c065d246bdfad3829;K8S_POD_UID=380d0df9-c4c5-4912-8a44-1cb19a3d8872\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4/380d0df9-c4c5-4912-8a44-1cb19a3d8872]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" Nov 01 09:22:25.337891 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:25.337845 5710 log.go:32] "RunPodSandbox from runtime service failed" err=< Nov 01 09:22:25.337891 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq_odh-ai-gateway-infra_f8adc733-8842-4850-a899-a95975c783e5_0(fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1" Netns:"/var/run/netns/efec40ab-4cf3-4905-a8fe-a5f5faba2558" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq;K8S_POD_INFRA_CONTAINER_ID=fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1;K8S_POD_UID=f8adc733-8842-4850-a899-a95975c783e5" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq/f8adc733-8842-4850-a899-a95975c783e5]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq?timeout=1m0s": context deadline exceeded Nov 01 09:22:25.337891 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:25.337891 ip-10-0-143-16 kubenswrapper[5710]: > Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:25.337909 5710 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq_odh-ai-gateway-infra_f8adc733-8842-4850-a899-a95975c783e5_0(fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1" Netns:"/var/run/netns/efec40ab-4cf3-4905-a8fe-a5f5faba2558" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq;K8S_POD_INFRA_CONTAINER_ID=fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1;K8S_POD_UID=f8adc733-8842-4850-a899-a95975c783e5" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq/f8adc733-8842-4850-a899-a95975c783e5]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq?timeout=1m0s": context deadline exceeded Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:25.337926 5710 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=< Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq_odh-ai-gateway-infra_f8adc733-8842-4850-a899-a95975c783e5_0(fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1" Netns:"/var/run/netns/efec40ab-4cf3-4905-a8fe-a5f5faba2558" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq;K8S_POD_INFRA_CONTAINER_ID=fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1;K8S_POD_UID=f8adc733-8842-4850-a899-a95975c783e5" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq/f8adc733-8842-4850-a899-a95975c783e5]: error waiting for pod: Get "https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq?timeout=1m0s": context deadline exceeded Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: > pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:25.338610 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:25.337997 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq_odh-ai-gateway-infra(f8adc733-8842-4850-a899-a95975c783e5)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq_odh-ai-gateway-infra(f8adc733-8842-4850-a899-a95975c783e5)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq_odh-ai-gateway-infra_f8adc733-8842-4850-a899-a95975c783e5_0(fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1): error adding pod odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1\\\" Netns:\\\"/var/run/netns/efec40ab-4cf3-4905-a8fe-a5f5faba2558\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq;K8S_POD_INFRA_CONTAINER_ID=fa44d3400ac4013e04a4134a98ecd8aeaf55a15de2b3b0d71b975c1711d51ed1;K8S_POD_UID=f8adc733-8842-4850-a899-a95975c783e5\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq/f8adc733-8842-4850-a899-a95975c783e5]: error waiting for pod: Get \\\"https://add429111eabe43cbaa078b9d9be88e3-5ea30804d4267b82.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" podUID="f8adc733-8842-4850-a899-a95975c783e5" Nov 01 09:22:25.730439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:25.730392 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:25.730845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:25.730826 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:26.159708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:26.159671 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq"] Nov 01 09:22:26.162521 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:22:26.162485 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8adc733_8842_4850_a899_a95975c783e5.slice/crio-29727cc62c31d141e0a7020f5b0c7244d9ab6714e9e24ba0db69e6ccf9e587f5 WatchSource:0}: Error finding container 29727cc62c31d141e0a7020f5b0c7244d9ab6714e9e24ba0db69e6ccf9e587f5: Status 404 returned error can't find the container with id 29727cc62c31d141e0a7020f5b0c7244d9ab6714e9e24ba0db69e6ccf9e587f5 Nov 01 09:22:26.164239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:26.164216 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:22:26.737452 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:26.737419 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" event={"ID":"f8adc733-8842-4850-a899-a95975c783e5","Type":"ContainerStarted","Data":"29727cc62c31d141e0a7020f5b0c7244d9ab6714e9e24ba0db69e6ccf9e587f5"} Nov 01 09:22:27.746027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:27.745982 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" event={"ID":"f8adc733-8842-4850-a899-a95975c783e5","Type":"ContainerStarted","Data":"782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9"} Nov 01 09:22:27.746584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:27.746098 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:27.869419 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:27.869348 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" podStartSLOduration=5.901537866 podStartE2EDuration="6.869325197s" totalImagesPullingTime="967.787331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:27.866193251 +0000 UTC m=+2970.954529924" watchObservedRunningTime="2026-11-01 09:22:27.869325197 +0000 UTC m=+2970.957661877" Nov 01 09:22:30.051242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:30.051188 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:30.051890 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:30.051871 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:30.463187 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:30.463157 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc"] Nov 01 09:22:30.464919 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:22:30.464888 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242da1e1_32b9_401a_9f7d_f42b00b17609.slice/crio-3e8b16e0021cbcffb4f746438baa30077c5c0e15cba4f8244c3e8226626e3997 WatchSource:0}: Error finding container 3e8b16e0021cbcffb4f746438baa30077c5c0e15cba4f8244c3e8226626e3997: Status 404 returned error can't find the container with id 3e8b16e0021cbcffb4f746438baa30077c5c0e15cba4f8244c3e8226626e3997 Nov 01 09:22:30.801120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:30.801010 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" event={"ID":"242da1e1-32b9-401a-9f7d-f42b00b17609","Type":"ContainerStarted","Data":"3e8b16e0021cbcffb4f746438baa30077c5c0e15cba4f8244c3e8226626e3997"} Nov 01 09:22:31.810318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:31.810224 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" event={"ID":"242da1e1-32b9-401a-9f7d-f42b00b17609","Type":"ContainerStarted","Data":"ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc"} Nov 01 09:22:31.810777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:31.810360 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:31.834947 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:31.834904 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:22:31.835113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:31.834987 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:22:31.920934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:31.920868 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" podStartSLOduration=36.04041585 podStartE2EDuration="36.920850434s" totalImagesPullingTime="880.434584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:31.920605589 +0000 UTC m=+2975.008942264" watchObservedRunningTime="2026-11-01 09:22:31.920850434 +0000 UTC m=+2975.009187107" Nov 01 09:22:32.825921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.825813 5710 generic.go:356] "Generic (PLEG): container finished" podID="9478fa58-762a-45cc-aa2f-91803a661f30" containerID="4cd84c1306ce8ebcdc041f515d4bd736e2874e5d2877a689375823ac6d588db1" exitCode=0 Nov 01 09:22:32.826831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.826795 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" event={"ID":"9478fa58-762a-45cc-aa2f-91803a661f30","Type":"ContainerDied","Data":"4cd84c1306ce8ebcdc041f515d4bd736e2874e5d2877a689375823ac6d588db1"} Nov 01 09:22:32.827295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.827275 5710 scope.go:122] "RemoveContainer" containerID="4cd84c1306ce8ebcdc041f515d4bd736e2874e5d2877a689375823ac6d588db1" Nov 01 09:22:32.829077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.829049 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:22:32.829150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.829126 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:22:32.829198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.829160 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:22:32.874645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.874608 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m"] Nov 01 09:22:32.880404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.880365 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:32.933919 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.933868 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m"] Nov 01 09:22:32.934472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.934449 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-e40e82\"" Nov 01 09:22:32.934697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:32.934592 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-e40e82-dockercfg-v8zvh\"" Nov 01 09:22:33.087681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.087543 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.088359 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.088334 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gjzj\" (UniqueName: \"kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj\") pod \"payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.227786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.227705 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.227958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.227935 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjzj\" (UniqueName: \"kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj\") pod \"payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.228461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.228441 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.286291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.286249 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjzj\" (UniqueName: \"kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj\") pod \"payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.512911 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.512869 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:33.758067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.758035 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:22:33.834688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.834595 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" event={"ID":"9478fa58-762a-45cc-aa2f-91803a661f30","Type":"ContainerStarted","Data":"280dd54cccfba196abf4a7311c2ce31039597f9fba59511445d235ee02ba98f5"} Nov 01 09:22:33.897154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:33.897074 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m"] Nov 01 09:22:33.938483 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:22:33.937467 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6dcfdd_f281_4ea4_9dbb_29b705b1c7b0.slice/crio-a4f6edf219dcb3833ec6028764902a7c3608ce0993a63ac9ecb9b50e5d0b901e WatchSource:0}: Error finding container a4f6edf219dcb3833ec6028764902a7c3608ce0993a63ac9ecb9b50e5d0b901e: Status 404 returned error can't find the container with id a4f6edf219dcb3833ec6028764902a7c3608ce0993a63ac9ecb9b50e5d0b901e Nov 01 09:22:34.804813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.804220 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:22:34.804813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.804372 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 09:22:34.804813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.804420 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 09:22:34.843105 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.843065 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" event={"ID":"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0","Type":"ContainerStarted","Data":"62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2"} Nov 01 09:22:34.843554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.843129 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" event={"ID":"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0","Type":"ContainerStarted","Data":"a4f6edf219dcb3833ec6028764902a7c3608ce0993a63ac9ecb9b50e5d0b901e"} Nov 01 09:22:34.843554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.843302 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:22:34.968087 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:34.968017 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" podStartSLOduration=2.967998488 podStartE2EDuration="2.967998488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:34.966292924 +0000 UTC m=+2978.054629596" watchObservedRunningTime="2026-11-01 09:22:34.967998488 +0000 UTC m=+2978.056335161" Nov 01 09:22:35.801271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:35.801228 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 09:22:35.801601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:35.801568 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 09:22:36.050935 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:36.050898 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:36.051367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:36.051269 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:36.456055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:36.456023 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4"] Nov 01 09:22:36.458662 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:22:36.458629 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380d0df9_c4c5_4912_8a44_1cb19a3d8872.slice/crio-1881f8579c26d022cfca6a38f30c550ec4b1330503644a30b3db5f70a454ec0c WatchSource:0}: Error finding container 1881f8579c26d022cfca6a38f30c550ec4b1330503644a30b3db5f70a454ec0c: Status 404 returned error can't find the container with id 1881f8579c26d022cfca6a38f30c550ec4b1330503644a30b3db5f70a454ec0c Nov 01 09:22:36.801808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:36.801266 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 09:22:36.801808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:36.801328 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 09:22:36.865977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:36.865938 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" event={"ID":"380d0df9-c4c5-4912-8a44-1cb19a3d8872","Type":"ContainerStarted","Data":"1881f8579c26d022cfca6a38f30c550ec4b1330503644a30b3db5f70a454ec0c"} Nov 01 09:22:37.811794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:37.809159 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 09:22:37.811794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:37.809219 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 09:22:37.841905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:37.841872 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:22:37.873796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:37.873759 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" event={"ID":"380d0df9-c4c5-4912-8a44-1cb19a3d8872","Type":"ContainerStarted","Data":"e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa"} Nov 01 09:22:37.873952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:37.873826 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:38.067121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:38.066999 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" podStartSLOduration=37.095108522 podStartE2EDuration="38.06697752s" totalImagesPullingTime="971.868998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:22:38.065774991 +0000 UTC m=+2981.154111657" watchObservedRunningTime="2026-11-01 09:22:38.06697752 +0000 UTC m=+2981.155314193" Nov 01 09:22:38.804222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:38.804173 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 09:22:38.805904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:38.805831 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 09:22:39.801260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:39.801209 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 01 09:22:39.802011 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:39.801959 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 01 09:22:40.805260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:40.805229 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:22:40.805760 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:40.805535 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:22:40.806293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:40.806275 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 09:22:43.888518 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:43.888489 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 09:22:45.479342 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:22:45.479306 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:22:45.862612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:22:45.862579 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:23:01.057210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:01.057180 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:23:01.065157 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:01.065130 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:23:01.453432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:01.453387 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:23:15.477837 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:23:15.477798 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:23:27.648021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.647974 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf"] Nov 01 09:23:27.652879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.652848 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.704226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.704174 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-58de5a-serving-cert\"" Nov 01 09:23:27.749352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.749303 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbt9h\" (UniqueName: \"kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.749560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.749361 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.749560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.749535 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.749710 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.749687 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.753569 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.753536 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf"] Nov 01 09:23:27.851530 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.851482 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.851802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.851587 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt9h\" (UniqueName: \"kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.851802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.851618 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.851802 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:23:27.851639 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-serving-cert: secret "maas-api-e2e-shared-a-w0-58de5a-serving-cert" not found Nov 01 09:23:27.851802 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:23:27.851709 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls podName:0b61e9c7-f611-4cfd-a401-66225c0d1d61 nodeName:}" failed. No retries permitted until 2026-11-01 09:23:28.351691619 +0000 UTC m=+3031.440028269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls") pod "maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" (UID: "0b61e9c7-f611-4cfd-a401-66225c0d1d61") : secret "maas-api-e2e-shared-a-w0-58de5a-serving-cert" not found Nov 01 09:23:27.852075 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.851814 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.852461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.852442 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.854644 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.854622 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:27.926801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:27.926557 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbt9h\" (UniqueName: \"kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:28.366073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:28.366036 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:28.368982 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:28.368947 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:28.568766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:28.568688 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:28.956702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:28.956676 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf"] Nov 01 09:23:28.959564 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:23:28.959532 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b61e9c7_f611_4cfd_a401_66225c0d1d61.slice/crio-934776c63e45704a2ec9e7edc7cdda789a36be50817add514fdb7f3c764df55d WatchSource:0}: Error finding container 934776c63e45704a2ec9e7edc7cdda789a36be50817add514fdb7f3c764df55d: Status 404 returned error can't find the container with id 934776c63e45704a2ec9e7edc7cdda789a36be50817add514fdb7f3c764df55d Nov 01 09:23:29.257535 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:29.257445 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" event={"ID":"0b61e9c7-f611-4cfd-a401-66225c0d1d61","Type":"ContainerStarted","Data":"934776c63e45704a2ec9e7edc7cdda789a36be50817add514fdb7f3c764df55d"} Nov 01 09:23:30.267121 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:30.267076 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" event={"ID":"0b61e9c7-f611-4cfd-a401-66225c0d1d61","Type":"ContainerStarted","Data":"bff47b3d7c138f3442d5157462326bea57c18087e1eb6ebacf4eb16f5c52ca86"} Nov 01 09:23:30.267692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:30.267162 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:30.378833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:30.378749 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" podStartSLOduration=2.528267907 podStartE2EDuration="3.378705278s" totalImagesPullingTime="850.437371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:23:30.378270032 +0000 UTC m=+3033.466606704" watchObservedRunningTime="2026-11-01 09:23:30.378705278 +0000 UTC m=+3033.467041947" Nov 01 09:23:32.831896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.831860 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln"] Nov 01 09:23:32.836397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.836369 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:32.886012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.885966 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-e40e82-serving-cert\"" Nov 01 09:23:32.935120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.935071 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln"] Nov 01 09:23:32.945519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.945479 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:32.945676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.945529 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:32.945676 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.945614 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqnfx\" (UniqueName: \"kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:32.945824 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:32.945804 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.046892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.046842 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.047178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.046909 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.047178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.046939 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqnfx\" (UniqueName: \"kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.047178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.047044 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.047178 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:23:33.047085 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-serving-cert: secret "maas-api-e2e-explicit-e40e82-serving-cert" not found Nov 01 09:23:33.047382 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:23:33.047226 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls podName:a558ea04-2b0b-4dd1-8df7-a1ac3268e094 nodeName:}" failed. No retries permitted until 2026-11-01 09:23:33.547187939 +0000 UTC m=+3036.635524606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls") pod "maas-api-e2e-explicit-e40e82-7f4566df-qqxln" (UID: "a558ea04-2b0b-4dd1-8df7-a1ac3268e094") : secret "maas-api-e2e-explicit-e40e82-serving-cert" not found Nov 01 09:23:33.047769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.047750 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.050369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.050336 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.123626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.119369 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqnfx\" (UniqueName: \"kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.563260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.563134 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.566016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.565991 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\") pod \"maas-api-e2e-explicit-e40e82-7f4566df-qqxln\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:33.754616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:33.754580 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:34.140277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:34.140245 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln"] Nov 01 09:23:34.143144 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:23:34.143106 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda558ea04_2b0b_4dd1_8df7_a1ac3268e094.slice/crio-ff8f5831dacd27aba395f44ecff06d971b783dcb39d64745e463dd771b138c8d WatchSource:0}: Error finding container ff8f5831dacd27aba395f44ecff06d971b783dcb39d64745e463dd771b138c8d: Status 404 returned error can't find the container with id ff8f5831dacd27aba395f44ecff06d971b783dcb39d64745e463dd771b138c8d Nov 01 09:23:34.304492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:34.304444 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" event={"ID":"a558ea04-2b0b-4dd1-8df7-a1ac3268e094","Type":"ContainerStarted","Data":"ff8f5831dacd27aba395f44ecff06d971b783dcb39d64745e463dd771b138c8d"} Nov 01 09:23:35.313111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:35.313000 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" event={"ID":"a558ea04-2b0b-4dd1-8df7-a1ac3268e094","Type":"ContainerStarted","Data":"54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a"} Nov 01 09:23:35.313111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:35.313078 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:35.429926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:35.429846 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" podStartSLOduration=2.565527006 podStartE2EDuration="3.429823617s" totalImagesPullingTime="864.296611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:23:35.426019157 +0000 UTC m=+3038.514355829" watchObservedRunningTime="2026-11-01 09:23:35.429823617 +0000 UTC m=+3038.518160288" Nov 01 09:23:36.279590 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:36.279556 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:23:41.326917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:41.326881 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:23:42.760297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:23:42.760250 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:23:45.478763 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:23:45.478703 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:24:14.862924 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.862884 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv"] Nov 01 09:24:14.868545 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.868520 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.925820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.925786 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.926033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.925940 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.926114 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.926044 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8pzk\" (UniqueName: \"kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.926227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.926208 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.926293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.926246 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.926386 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.926372 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:14.964299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.964266 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"kube-root-ca.crt\"" Nov 01 09:24:14.964299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.964265 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"openshift-service-ca.crt\"" Nov 01 09:24:14.964600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.964376 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"auth-test-model-fd5a10-kserve-self-signed-certs\"" Nov 01 09:24:14.964600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.964430 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fd5a10\"/\"default-dockercfg-q8jh9\"" Nov 01 09:24:14.967355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:14.967326 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv"] Nov 01 09:24:15.038819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.038778 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.040924 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.040889 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.041734 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.041686 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.042349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.042226 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.042629 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.042611 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8pzk\" (UniqueName: \"kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.050578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.050549 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.051219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.051186 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.051895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.050165 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.052097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.051113 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.052228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.051651 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.052228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.050330 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.100354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.100323 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8pzk\" (UniqueName: \"kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk\") pod \"auth-test-model-fd5a10-kserve-6cf5749857-r5ksv\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.181654 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.181548 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:15.478627 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:24:15.478538 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:24:15.566303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.566264 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv"] Nov 01 09:24:15.568627 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:24:15.568592 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d0d061f_2fcf_4ac3_882c_1bc7af451253.slice/crio-23a585ad87960fd9cb5339311f249dbf03a4210b37a95599dd485bac4bd8e1d4 WatchSource:0}: Error finding container 23a585ad87960fd9cb5339311f249dbf03a4210b37a95599dd485bac4bd8e1d4: Status 404 returned error can't find the container with id 23a585ad87960fd9cb5339311f249dbf03a4210b37a95599dd485bac4bd8e1d4 Nov 01 09:24:15.590844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:15.590809 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" event={"ID":"3d0d061f-2fcf-4ac3-882c-1bc7af451253","Type":"ContainerStarted","Data":"23a585ad87960fd9cb5339311f249dbf03a4210b37a95599dd485bac4bd8e1d4"} Nov 01 09:24:16.342777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:16.342740 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:24:16.599293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:16.599192 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" event={"ID":"3d0d061f-2fcf-4ac3-882c-1bc7af451253","Type":"ContainerStarted","Data":"ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89"} Nov 01 09:24:16.712869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:16.712799 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" podStartSLOduration=2.712776834 podStartE2EDuration="2.712776834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:24:16.708843354 +0000 UTC m=+3079.797180027" watchObservedRunningTime="2026-11-01 09:24:16.712776834 +0000 UTC m=+3079.801113511" Nov 01 09:24:17.181889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:17.181853 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:25.182016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:25.181975 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:25.195411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:25.195377 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:25.682943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:25.682905 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:24:44.829102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.829059 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd"] Nov 01 09:24:44.834094 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.834067 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.903897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.903854 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd"] Nov 01 09:24:44.919628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.919594 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.919892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.919655 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.919892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.919689 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.919892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.919859 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.920087 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.919932 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.920087 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.920037 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.920208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.920142 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.920208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.920182 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbx8h\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.920327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.920207 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:44.944084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:44.944040 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a83714-openshift-default-dockercfg-6fhhb\"" Nov 01 09:24:45.022044 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022001 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022044 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022048 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022134 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022239 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022268 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022320 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022365 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx8h\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022394 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022498 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022635 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.022859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.022667 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.023337 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.023287 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.023472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.023426 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.023472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.023450 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.025269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.025244 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.025384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.025255 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.200740 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.199591 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.205682 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.205644 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx8h\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h\") pod \"e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.449581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.449547 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:45.480551 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:24:45.480509 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:24:45.884658 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.884619 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd"] Nov 01 09:24:45.886460 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:24:45.886425 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182f210f_0aa2_44bb_b7c7_84d04871716e.slice/crio-a80644a64d41d531c39aea1939ee3440c57969b6b98b67f9f664260e7de62609 WatchSource:0}: Error finding container a80644a64d41d531c39aea1939ee3440c57969b6b98b67f9f664260e7de62609: Status 404 returned error can't find the container with id a80644a64d41d531c39aea1939ee3440c57969b6b98b67f9f664260e7de62609 Nov 01 09:24:45.889600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.889546 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:24:45.889710 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.889637 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:24:45.889710 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:45.889666 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:24:46.832788 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:46.832707 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" event={"ID":"182f210f-0aa2-44bb-b7c7-84d04871716e","Type":"ContainerStarted","Data":"e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9"} Nov 01 09:24:46.833068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:46.832811 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" event={"ID":"182f210f-0aa2-44bb-b7c7-84d04871716e","Type":"ContainerStarted","Data":"a80644a64d41d531c39aea1939ee3440c57969b6b98b67f9f664260e7de62609"} Nov 01 09:24:46.994049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:46.993983 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podStartSLOduration=2.993962353 podStartE2EDuration="2.993962353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:24:46.992543014 +0000 UTC m=+3110.080879686" watchObservedRunningTime="2026-11-01 09:24:46.993962353 +0000 UTC m=+3110.082299026" Nov 01 09:24:47.449801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:47.449765 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:47.455906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:47.455872 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:47.840274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:47.840240 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:47.841407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:47.841382 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:24:51.152347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:51.152290 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:24:53.586241 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.586181 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z"] Nov 01 09:24:53.620043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.609044 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.633300 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.633264 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z"] Nov 01 09:24:53.673453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.673410 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a83714\"" Nov 01 09:24:53.673670 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.673488 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a83714-dockercfg-gv6zm\"" Nov 01 09:24:53.700196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.700161 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87j66\" (UniqueName: \"kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66\") pod \"payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.700412 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.700389 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.801370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.801323 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.801606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.801401 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87j66\" (UniqueName: \"kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66\") pod \"payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.802144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.802117 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.859349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.859298 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87j66\" (UniqueName: \"kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66\") pod \"payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:53.964470 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:53.964424 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:54.347694 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:54.347665 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z"] Nov 01 09:24:54.350514 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:24:54.350478 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6be58b_baed_445c_a199_aee6ae24314a.slice/crio-0c56261b04e56f5256f13b9ee036f72502359cfc7b178b5f8fac7c06bc0e2bce WatchSource:0}: Error finding container 0c56261b04e56f5256f13b9ee036f72502359cfc7b178b5f8fac7c06bc0e2bce: Status 404 returned error can't find the container with id 0c56261b04e56f5256f13b9ee036f72502359cfc7b178b5f8fac7c06bc0e2bce Nov 01 09:24:54.889608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:54.889575 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" event={"ID":"5d6be58b-baed-445c-a199-aee6ae24314a","Type":"ContainerStarted","Data":"ac037825b4ee9ad7f0bbfeff435a38738f0c17ee6733afea54d44ab0c0e15925"} Nov 01 09:24:54.890051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:54.889628 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" event={"ID":"5d6be58b-baed-445c-a199-aee6ae24314a","Type":"ContainerStarted","Data":"0c56261b04e56f5256f13b9ee036f72502359cfc7b178b5f8fac7c06bc0e2bce"} Nov 01 09:24:54.890051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:54.889700 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:24:55.000313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:55.000241 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" podStartSLOduration=2.00022582 podStartE2EDuration="2.00022582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:24:54.997913549 +0000 UTC m=+3118.086250222" watchObservedRunningTime="2026-11-01 09:24:55.00022582 +0000 UTC m=+3118.088562492" Nov 01 09:24:55.802641 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:55.802604 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks"] Nov 01 09:24:55.802917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:55.802889 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" containerID="cri-o://9c16c98ea5e1f81db6c089041cb6aff6985738e906813e72d71e29f413e09cb4" gracePeriod=30 Nov 01 09:24:55.969204 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:55.969163 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb"] Nov 01 09:24:55.969743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:24:55.969417 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" containerID="cri-o://3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed" gracePeriod=30 Nov 01 09:25:00.883015 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:00.882974 5710 patch_prober.go:29] interesting pod/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:25:00.883536 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:00.883043 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:25:00.936381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:00.936348 5710 generic.go:356] "Generic (PLEG): container finished" podID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerID="9c16c98ea5e1f81db6c089041cb6aff6985738e906813e72d71e29f413e09cb4" exitCode=0 Nov 01 09:25:00.936563 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:00.936423 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" event={"ID":"18480453-3419-4ee8-8d6e-6af8aa27537e","Type":"ContainerDied","Data":"9c16c98ea5e1f81db6c089041cb6aff6985738e906813e72d71e29f413e09cb4"} Nov 01 09:25:01.100149 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.100119 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:25:01.111780 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.111712 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.111817 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.111858 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.111930 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.111953 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.111968 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.112310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112035 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112075 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112176 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112212 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data\") pod \"18480453-3419-4ee8-8d6e-6af8aa27537e\" (UID: \"18480453-3419-4ee8-8d6e-6af8aa27537e\") " Nov 01 09:25:01.112529 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112313 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.112529 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112253 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:25:01.112659 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112592 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.112659 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112629 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.112977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112959 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.113055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.112980 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18480453-3419-4ee8-8d6e-6af8aa27537e-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.113055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.113006 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.113055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.113016 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.113055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.113025 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.115019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.114990 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.115169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.115046 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "kube-api-access-z46b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:25:01.115251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.115231 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:25:01.115251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.115235 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token" pod "18480453-3419-4ee8-8d6e-6af8aa27537e" (UID: "18480453-3419-4ee8-8d6e-6af8aa27537e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:25:01.214554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.214512 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.214554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.214559 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.214812 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.214576 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.214812 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.214609 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46b6\" (UniqueName: \"kubernetes.io/projected/18480453-3419-4ee8-8d6e-6af8aa27537e-kube-api-access-z46b6\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.252616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.252586 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:25:01.315792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.315757 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.315792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.315802 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.315821 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.315846 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.315882 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316014 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316078 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316115 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316227 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket\") pod \"ea95f58f-1bc9-41c2-885a-d7f90b862454\" (UID: \"ea95f58f-1bc9-41c2-885a-d7f90b862454\") " Nov 01 09:25:01.316503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316378 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.316503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316410 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:25:01.316696 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316611 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.316910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.316777 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.317131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.317106 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.317923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.317892 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea95f58f-1bc9-41c2-885a-d7f90b862454-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.317923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.317925 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.318133 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.317942 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.318133 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.317959 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.318133 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.317974 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.318513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.318491 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "kube-api-access-rzrvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:25:01.318513 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.318495 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:25:01.318600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.318502 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:25:01.318600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.318570 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo" pod "ea95f58f-1bc9-41c2-885a-d7f90b862454" (UID: "ea95f58f-1bc9-41c2-885a-d7f90b862454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:25:01.419754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.419680 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.419754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.419711 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.419754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.419744 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea95f58f-1bc9-41c2-885a-d7f90b862454-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.419754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.419757 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzrvk\" (UniqueName: \"kubernetes.io/projected/ea95f58f-1bc9-41c2-885a-d7f90b862454-kube-api-access-rzrvk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:25:01.947165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.947132 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" Nov 01 09:25:01.947165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.947153 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks" event={"ID":"18480453-3419-4ee8-8d6e-6af8aa27537e","Type":"ContainerDied","Data":"e1836339632fb234a827bbe383924ce1d16c6af9dd40d1d5ac55f3c59c09d586"} Nov 01 09:25:01.948202 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.947233 5710 scope.go:122] "RemoveContainer" containerID="9c16c98ea5e1f81db6c089041cb6aff6985738e906813e72d71e29f413e09cb4" Nov 01 09:25:01.949437 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.949407 5710 generic.go:356] "Generic (PLEG): container finished" podID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerID="3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed" exitCode=0 Nov 01 09:25:01.949553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.949468 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" Nov 01 09:25:01.949553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.949488 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" event={"ID":"ea95f58f-1bc9-41c2-885a-d7f90b862454","Type":"ContainerDied","Data":"3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed"} Nov 01 09:25:01.949553 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.949544 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb" event={"ID":"ea95f58f-1bc9-41c2-885a-d7f90b862454","Type":"ContainerDied","Data":"8f9022c6758ccc27f2a7ed18e8071a1f31125bb05afbe3f2d167a8b7c1943d3c"} Nov 01 09:25:01.958631 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.958611 5710 scope.go:122] "RemoveContainer" containerID="3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed" Nov 01 09:25:01.969762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.969735 5710 scope.go:122] "RemoveContainer" containerID="3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed" Nov 01 09:25:01.970111 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:25:01.970086 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed\": container with ID starting with 3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed not found: ID does not exist" containerID="3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed" Nov 01 09:25:01.970166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:01.970125 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed"} err="failed to get container status \"3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed\": rpc error: code = NotFound desc = could not find container \"3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed\": container with ID starting with 3a7349ab09d0638008bb957918041b27622deb8cf54bed2c9de87db861b332ed not found: ID does not exist" Nov 01 09:25:02.150353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:02.150304 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb"] Nov 01 09:25:02.196194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:02.196159 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-65hmb"] Nov 01 09:25:02.256271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:02.256233 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks"] Nov 01 09:25:02.260938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:02.260905 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-2c7ks"] Nov 01 09:25:03.059766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:03.059733 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" path="/var/lib/kubelet/pods/18480453-3419-4ee8-8d6e-6af8aa27537e/volumes" Nov 01 09:25:03.060366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:03.060350 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" path="/var/lib/kubelet/pods/ea95f58f-1bc9-41c2-885a-d7f90b862454/volumes" Nov 01 09:25:05.901142 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:05.901109 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:25:15.480037 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:25:15.479999 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:25:25.705181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:25.705138 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:25:45.478472 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:25:45.478386 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:25:58.617278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.617242 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg"] Nov 01 09:25:58.617930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.617914 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" Nov 01 09:25:58.617930 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.617930 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" Nov 01 09:25:58.618025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.617948 5710 cpu_manager.go:414] "containerMap: removing container" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" Nov 01 09:25:58.618025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.617953 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" Nov 01 09:25:58.618090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.618064 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea95f58f-1bc9-41c2-885a-d7f90b862454" containerName="istio-proxy" Nov 01 09:25:58.618090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.618073 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="18480453-3419-4ee8-8d6e-6af8aa27537e" containerName="istio-proxy" Nov 01 09:25:58.621668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.621639 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.672905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.672846 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-a83714-serving-cert\"" Nov 01 09:25:58.722211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.722172 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg"] Nov 01 09:25:58.730735 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.730685 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb5q\" (UniqueName: \"kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.730931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.730746 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.730931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.730855 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.731045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.731032 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.831906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.831858 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.832140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.832019 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.832140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.832119 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb5q\" (UniqueName: \"kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.832268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.832142 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.832268 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:25:58.832200 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-serving-cert: secret "maas-api-e2e-shared-b-w0-a83714-serving-cert" not found Nov 01 09:25:58.832379 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:25:58.832284 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls podName:28237a9a-dc8b-4c73-957f-41e47e14689c nodeName:}" failed. No retries permitted until 2026-11-01 09:25:59.332258993 +0000 UTC m=+3182.420595659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls") pod "maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" (UID: "28237a9a-dc8b-4c73-957f-41e47e14689c") : secret "maas-api-e2e-shared-b-w0-a83714-serving-cert" not found Nov 01 09:25:58.832946 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.832925 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.834749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.834689 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:58.901642 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:58.901558 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb5q\" (UniqueName: \"kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:59.340651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:59.340612 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:59.343493 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:59.343458 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:59.538497 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:59.538454 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:25:59.918589 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:25:59.918555 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg"] Nov 01 09:25:59.921349 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:25:59.921316 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28237a9a_dc8b_4c73_957f_41e47e14689c.slice/crio-bc6f8429858999ef00e9c3f6001663a592efb1ddc4ca413be96c8def7f27f737 WatchSource:0}: Error finding container bc6f8429858999ef00e9c3f6001663a592efb1ddc4ca413be96c8def7f27f737: Status 404 returned error can't find the container with id bc6f8429858999ef00e9c3f6001663a592efb1ddc4ca413be96c8def7f27f737 Nov 01 09:26:00.414645 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:00.414608 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" event={"ID":"28237a9a-dc8b-4c73-957f-41e47e14689c","Type":"ContainerStarted","Data":"bc6f8429858999ef00e9c3f6001663a592efb1ddc4ca413be96c8def7f27f737"} Nov 01 09:26:01.422537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:01.422493 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" event={"ID":"28237a9a-dc8b-4c73-957f-41e47e14689c","Type":"ContainerStarted","Data":"bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5"} Nov 01 09:26:01.423001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:01.422568 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:26:01.536226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:01.536159 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" podStartSLOduration=2.577513154 podStartE2EDuration="3.536137913s" totalImagesPullingTime="958.624759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:26:01.533345926 +0000 UTC m=+3184.621682609" watchObservedRunningTime="2026-11-01 09:26:01.536137913 +0000 UTC m=+3184.624474591" Nov 01 09:26:07.435483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:07.435448 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:26:15.480993 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:26:15.480946 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:26:21.678073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:21.678033 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:26:32.677573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.677540 5710 generic.go:356] "Generic (PLEG): container finished" podID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerID="e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9" exitCode=0 Nov 01 09:26:32.678058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.677593 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" event={"ID":"182f210f-0aa2-44bb-b7c7-84d04871716e","Type":"ContainerDied","Data":"e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9"} Nov 01 09:26:32.678058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.678025 5710 scope.go:122] "RemoveContainer" containerID="e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9" Nov 01 09:26:32.679840 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.679802 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:26:32.679970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.679899 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:26:32.679970 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.679946 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:26:32.840759 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:32.840688 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:26:33.710934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:33.710879 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" event={"ID":"182f210f-0aa2-44bb-b7c7-84d04871716e","Type":"ContainerStarted","Data":"703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1"} Nov 01 09:26:34.450560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:34.450522 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:26:34.451887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:34.451857 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 09:26:34.452043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:34.451922 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 09:26:35.450083 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:35.450044 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 09:26:35.450498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:35.450109 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 09:26:36.450612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:36.450578 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 09:26:36.451021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:36.450635 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 09:26:37.451013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:37.450974 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 09:26:37.451461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:37.451033 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 09:26:38.450856 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:38.450818 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 01 09:26:38.451054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:38.450880 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 01 09:26:39.454883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:39.454852 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:26:39.455315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:39.455118 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:26:39.455948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:39.455926 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:26:45.483022 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:26:45.482980 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:26:55.184618 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:26:55.184577 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m"] Nov 01 09:27:00.185356 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.185303 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" podUID="1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" containerName="payload-pre-processing" containerID="cri-o://62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2" gracePeriod=25 Nov 01 09:27:00.471102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.471066 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:27:00.505024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.504985 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj\" (UniqueName: \"kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj\") pod \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " Nov 01 09:27:00.505216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.505198 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume\") pod \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\" (UID: \"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0\") " Nov 01 09:27:00.505570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.505546 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume" pod "1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" (UID: "1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:27:00.505873 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.505690 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:00.507327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.507304 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj" pod "1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" (UID: "1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0"). InnerVolumeSpecName "kube-api-access-4gjzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:27:00.607085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.607051 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gjzj\" (UniqueName: \"kubernetes.io/projected/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0-kube-api-access-4gjzj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:00.929688 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.929648 5710 generic.go:356] "Generic (PLEG): container finished" podID="1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" containerID="62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2" exitCode=0 Nov 01 09:27:00.929929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.929703 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" event={"ID":"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0","Type":"ContainerDied","Data":"62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2"} Nov 01 09:27:00.929929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.929700 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" Nov 01 09:27:00.929929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.929786 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m" event={"ID":"1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0","Type":"ContainerDied","Data":"a4f6edf219dcb3833ec6028764902a7c3608ce0993a63ac9ecb9b50e5d0b901e"} Nov 01 09:27:00.929929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.929810 5710 scope.go:122] "RemoveContainer" containerID="62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2" Nov 01 09:27:00.941217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.941200 5710 scope.go:122] "RemoveContainer" containerID="62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2" Nov 01 09:27:00.941567 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:27:00.941547 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2\": container with ID starting with 62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2 not found: ID does not exist" containerID="62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2" Nov 01 09:27:00.941630 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:00.941576 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2"} err="failed to get container status \"62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2\": rpc error: code = NotFound desc = could not find container \"62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2\": container with ID starting with 62e46d39f3ae2b8039a042847e0da470594717d4ed7ffdb11afece87f720b7d2 not found: ID does not exist" Nov 01 09:27:01.085137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.085090 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m"] Nov 01 09:27:01.131385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.131350 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e40e82-795695b587-xct7m"] Nov 01 09:27:01.291628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.291520 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln"] Nov 01 09:27:01.292176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.291859 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" podUID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" containerName="maas-api" containerID="cri-o://54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a" gracePeriod=30 Nov 01 09:27:01.321109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.321068 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" podUID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.210:8443/health\": dial tcp 10.134.0.210:8443: connect: connection refused" Nov 01 09:27:01.635106 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.635077 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:27:01.727692 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.727653 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\") pod \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " Nov 01 09:27:01.732423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.731779 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca\") pod \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " Nov 01 09:27:01.732423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.731831 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls" pod "a558ea04-2b0b-4dd1-8df7-a1ac3268e094" (UID: "a558ea04-2b0b-4dd1-8df7-a1ac3268e094"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:27:01.732423 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.732224 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca" pod "a558ea04-2b0b-4dd1-8df7-a1ac3268e094" (UID: "a558ea04-2b0b-4dd1-8df7-a1ac3268e094"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:27:01.733867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.732470 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs\") pod \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " Nov 01 09:27:01.733867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.732912 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx\" (UniqueName: \"kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx\") pod \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\" (UID: \"a558ea04-2b0b-4dd1-8df7-a1ac3268e094\") " Nov 01 09:27:01.733867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.733601 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:01.733867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.733639 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:01.736621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.736590 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx" pod "a558ea04-2b0b-4dd1-8df7-a1ac3268e094" (UID: "a558ea04-2b0b-4dd1-8df7-a1ac3268e094"). InnerVolumeSpecName "kube-api-access-wqnfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:27:01.737176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.737152 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs" pod "a558ea04-2b0b-4dd1-8df7-a1ac3268e094" (UID: "a558ea04-2b0b-4dd1-8df7-a1ac3268e094"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:27:01.834581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.834532 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:01.834581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.834571 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqnfx\" (UniqueName: \"kubernetes.io/projected/a558ea04-2b0b-4dd1-8df7-a1ac3268e094-kube-api-access-wqnfx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:01.943122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.943022 5710 generic.go:356] "Generic (PLEG): container finished" podID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" containerID="54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a" exitCode=0 Nov 01 09:27:01.943122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.943074 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" Nov 01 09:27:01.943122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.943098 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" event={"ID":"a558ea04-2b0b-4dd1-8df7-a1ac3268e094","Type":"ContainerDied","Data":"54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a"} Nov 01 09:27:01.943488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.943151 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln" event={"ID":"a558ea04-2b0b-4dd1-8df7-a1ac3268e094","Type":"ContainerDied","Data":"ff8f5831dacd27aba395f44ecff06d971b783dcb39d64745e463dd771b138c8d"} Nov 01 09:27:01.943488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.943173 5710 scope.go:122] "RemoveContainer" containerID="54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a" Nov 01 09:27:01.957020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.956937 5710 scope.go:122] "RemoveContainer" containerID="54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a" Nov 01 09:27:01.957438 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:27:01.957416 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a\": container with ID starting with 54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a not found: ID does not exist" containerID="54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a" Nov 01 09:27:01.957577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:01.957446 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a"} err="failed to get container status \"54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a\": rpc error: code = NotFound desc = could not find container \"54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a\": container with ID starting with 54bd307f2100bafbe05542fc16e543a4acf46fcfce0adb3e53a8e424d9086c2a not found: ID does not exist" Nov 01 09:27:02.100000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:02.099963 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln"] Nov 01 09:27:02.102751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:02.102709 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e40e82-7f4566df-qqxln"] Nov 01 09:27:03.060613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:03.060573 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" path="/var/lib/kubelet/pods/1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0/volumes" Nov 01 09:27:03.061132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:03.061116 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" path="/var/lib/kubelet/pods/a558ea04-2b0b-4dd1-8df7-a1ac3268e094/volumes" Nov 01 09:27:15.478585 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:27:15.478544 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:27:23.204652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:23.204612 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:27:45.480563 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:27:45.480524 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:27:48.025616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:48.025577 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8"] Nov 01 09:27:48.026175 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:48.025917 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" containerID="cri-o://393eb08c095a2d51c5d517c860c268ae331b1e3fd4dc0ab3bcf625b37532f9e1" gracePeriod=30 Nov 01 09:27:53.183589 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.183541 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 01 09:27:53.184102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.183619 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 01 09:27:53.290346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.290272 5710 generic.go:356] "Generic (PLEG): container finished" podID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerID="393eb08c095a2d51c5d517c860c268ae331b1e3fd4dc0ab3bcf625b37532f9e1" exitCode=0 Nov 01 09:27:53.290514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.290345 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" event={"ID":"131e4285-dd5e-4eea-9449-596a1aac2c21","Type":"ContainerDied","Data":"393eb08c095a2d51c5d517c860c268ae331b1e3fd4dc0ab3bcf625b37532f9e1"} Nov 01 09:27:53.290514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.290402 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" event={"ID":"131e4285-dd5e-4eea-9449-596a1aac2c21","Type":"ContainerDied","Data":"1803e029c733b00d7aca6cb91b3b2803b7c7e9609e2e9947b8eeeacddd0fa045"} Nov 01 09:27:53.290514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.290421 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1803e029c733b00d7aca6cb91b3b2803b7c7e9609e2e9947b8eeeacddd0fa045" Nov 01 09:27:53.313968 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.313941 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:27:53.385963 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.385911 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386125 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386216 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386262 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386285 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:27:53.386410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386311 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386457 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386538 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386486 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386534 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386540 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:27:53.386757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386613 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy\") pod \"131e4285-dd5e-4eea-9449-596a1aac2c21\" (UID: \"131e4285-dd5e-4eea-9449-596a1aac2c21\") " Nov 01 09:27:53.386757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386709 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:27:53.386887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386771 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:27:53.386951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.386907 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:27:53.387283 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.387260 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.387384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.387287 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.387384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.387302 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131e4285-dd5e-4eea-9449-596a1aac2c21-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.387384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.387315 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.387384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.387327 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.388852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.388825 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:27:53.389022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.389003 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:27:53.389268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.389243 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:27:53.389268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.389253 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq" pod "131e4285-dd5e-4eea-9449-596a1aac2c21" (UID: "131e4285-dd5e-4eea-9449-596a1aac2c21"). InnerVolumeSpecName "kube-api-access-s45bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:27:53.489585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.489549 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.489585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.489579 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.489585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.489589 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s45bq\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-kube-api-access-s45bq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:53.489902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:53.489600 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131e4285-dd5e-4eea-9449-596a1aac2c21-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:27:54.306750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:54.303521 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8" Nov 01 09:27:54.518564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:54.518526 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8"] Nov 01 09:27:54.564459 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:54.564358 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-2fvm8"] Nov 01 09:27:55.059285 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:27:55.059249 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" path="/var/lib/kubelet/pods/131e4285-dd5e-4eea-9449-596a1aac2c21/volumes" Nov 01 09:28:00.288837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:28:00.288802 5710 scope.go:122] "RemoveContainer" containerID="393eb08c095a2d51c5d517c860c268ae331b1e3fd4dc0ab3bcf625b37532f9e1" Nov 01 09:28:01.100842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:28:01.100809 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:28:01.109528 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:28:01.109503 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:28:15.478123 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:28:15.478081 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:28:16.112879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:28:16.112840 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:28:45.482059 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:28:45.481967 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:29:09.306812 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:29:09.306769 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:29:15.478941 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:29:15.478899 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:29:45.486093 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:29:45.486054 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:29:58.054222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:29:58.054167 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt"] Nov 01 09:29:58.054908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:29:58.054397 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" containerID="cri-o://a61e73004f5ca18fb995adcfd03b9a8d1c9d23d2e25c8dea85d1e3b63726eb47" gracePeriod=30 Nov 01 09:29:58.243584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:29:58.243538 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r"] Nov 01 09:29:58.243888 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:29:58.243826 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" containerID="cri-o://a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2" gracePeriod=30 Nov 01 09:30:01.642069 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:01.642034 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:30:01.642498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:01.642090 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:30:02.746064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:02.746012 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:30:03.211118 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.211078 5710 generic.go:356] "Generic (PLEG): container finished" podID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerID="a61e73004f5ca18fb995adcfd03b9a8d1c9d23d2e25c8dea85d1e3b63726eb47" exitCode=0 Nov 01 09:30:03.211372 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.211122 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" event={"ID":"35a9ac32-c025-4ab8-8387-89f8bcdeedef","Type":"ContainerDied","Data":"a61e73004f5ca18fb995adcfd03b9a8d1c9d23d2e25c8dea85d1e3b63726eb47"} Nov 01 09:30:03.379062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.379036 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428216 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428284 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428363 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428582 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428639 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428799 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428859 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428902 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.428977 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy\") pod \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\" (UID: \"35a9ac32-c025-4ab8-8387-89f8bcdeedef\") " Nov 01 09:30:03.438766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.438296 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:30:03.441562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.441223 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.445523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.444512 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.445523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.444655 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.445523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.445393 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:30:03.445523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.445498 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:30:03.448948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.448904 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.449809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.449666 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.450132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.450105 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg" pod "35a9ac32-c025-4ab8-8387-89f8bcdeedef" (UID: "35a9ac32-c025-4ab8-8387-89f8bcdeedef"). InnerVolumeSpecName "kube-api-access-zd7sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:30:03.531766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531694 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.531766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531767 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531782 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531798 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7sg\" (UniqueName: \"kubernetes.io/projected/35a9ac32-c025-4ab8-8387-89f8bcdeedef-kube-api-access-zd7sg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531811 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531823 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531835 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531847 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.532053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.531858 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ac32-c025-4ab8-8387-89f8bcdeedef-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.558652 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.558620 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:30:03.632955 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.632853 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.632955 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.632949 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633033 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633074 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633121 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633169 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633275 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633411 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633500 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token\") pod \"c3fa8130-1812-40fe-82ff-5aa990333199\" (UID: \"c3fa8130-1812-40fe-82ff-5aa990333199\") " Nov 01 09:30:03.633972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.633938 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.634165 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.634141 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.634293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.634160 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.634561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.634371 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.635112 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.635077 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:30:03.641924 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.641873 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:30:03.649291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.649211 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "kube-api-access-d7lhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:30:03.651984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.651945 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.662104 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.662059 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:30:03.664901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.664844 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy" pod "c3fa8130-1812-40fe-82ff-5aa990333199" (UID: "c3fa8130-1812-40fe-82ff-5aa990333199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:03.739712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739669 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.739712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739706 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fa8130-1812-40fe-82ff-5aa990333199-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.739712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739738 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.739712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739752 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fa8130-1812-40fe-82ff-5aa990333199-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.739712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739765 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7lhf\" (UniqueName: \"kubernetes.io/projected/c3fa8130-1812-40fe-82ff-5aa990333199-kube-api-access-d7lhf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.740502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739783 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.740502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739797 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:03.740502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:03.739809 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fa8130-1812-40fe-82ff-5aa990333199-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:04.222810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.222691 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" Nov 01 09:30:04.224262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.222695 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt" event={"ID":"35a9ac32-c025-4ab8-8387-89f8bcdeedef","Type":"ContainerDied","Data":"b783cb25de63ca9168e6d44b368779b2a2eca18bf9aa9af2d2fe773e3cc8a72a"} Nov 01 09:30:04.224262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.222986 5710 scope.go:122] "RemoveContainer" containerID="a61e73004f5ca18fb995adcfd03b9a8d1c9d23d2e25c8dea85d1e3b63726eb47" Nov 01 09:30:04.225180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.225155 5710 generic.go:356] "Generic (PLEG): container finished" podID="c3fa8130-1812-40fe-82ff-5aa990333199" containerID="a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2" exitCode=0 Nov 01 09:30:04.225320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.225246 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" event={"ID":"c3fa8130-1812-40fe-82ff-5aa990333199","Type":"ContainerDied","Data":"a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2"} Nov 01 09:30:04.225320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.225295 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" event={"ID":"c3fa8130-1812-40fe-82ff-5aa990333199","Type":"ContainerDied","Data":"a392737dcafaca501779f554e230b98c8245d73a0b1ab70dcf31b46b73ff1f92"} Nov 01 09:30:04.225443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.225367 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r" Nov 01 09:30:04.236597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.236574 5710 scope.go:122] "RemoveContainer" containerID="a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2" Nov 01 09:30:04.249622 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.249602 5710 scope.go:122] "RemoveContainer" containerID="a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2" Nov 01 09:30:04.249988 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:30:04.249967 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2\": container with ID starting with a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2 not found: ID does not exist" containerID="a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2" Nov 01 09:30:04.250045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.250000 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2"} err="failed to get container status \"a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2\": rpc error: code = NotFound desc = could not find container \"a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2\": container with ID starting with a7512f43d2e59c4ca692bb33f78b5829994104b5f109ca969056bc6b84b7b4e2 not found: ID does not exist" Nov 01 09:30:04.378999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.378901 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r"] Nov 01 09:30:04.425269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.425231 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-2656r"] Nov 01 09:30:04.487782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.487744 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt"] Nov 01 09:30:04.493571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:04.493542 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-88kgt"] Nov 01 09:30:05.059639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:05.059603 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" path="/var/lib/kubelet/pods/35a9ac32-c025-4ab8-8387-89f8bcdeedef/volumes" Nov 01 09:30:05.060139 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:05.060122 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" path="/var/lib/kubelet/pods/c3fa8130-1812-40fe-82ff-5aa990333199/volumes" Nov 01 09:30:07.928923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.928885 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr"] Nov 01 09:30:07.929985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.929961 5710 cpu_manager.go:414] "containerMap: removing container" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" Nov 01 09:30:07.930047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.929989 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" Nov 01 09:30:07.930086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930046 5710 cpu_manager.go:414] "containerMap: removing container" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" Nov 01 09:30:07.930086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930054 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" Nov 01 09:30:07.930086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930079 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" Nov 01 09:30:07.930188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930087 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" Nov 01 09:30:07.930188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930103 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" containerName="payload-pre-processing" Nov 01 09:30:07.930188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930110 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" containerName="payload-pre-processing" Nov 01 09:30:07.930188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930146 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" containerName="maas-api" Nov 01 09:30:07.930188 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930153 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" containerName="maas-api" Nov 01 09:30:07.930353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930341 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a558ea04-2b0b-4dd1-8df7-a1ac3268e094" containerName="maas-api" Nov 01 09:30:07.930388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930363 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="131e4285-dd5e-4eea-9449-596a1aac2c21" containerName="istio-proxy" Nov 01 09:30:07.930388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930380 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a9ac32-c025-4ab8-8387-89f8bcdeedef" containerName="istio-proxy" Nov 01 09:30:07.930489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930393 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fa8130-1812-40fe-82ff-5aa990333199" containerName="istio-proxy" Nov 01 09:30:07.930489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.930404 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6dcfdd-f281-4ea4-9dbb-29b705b1c7b0" containerName="payload-pre-processing" Nov 01 09:30:07.935556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.935528 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:07.986487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.986449 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-54c25e-serving-cert\"" Nov 01 09:30:07.994523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.994481 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:07.994711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.994635 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qw7\" (UniqueName: \"kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:07.994874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.994856 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:07.994954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:07.994928 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.036020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.035982 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr"] Nov 01 09:30:08.095993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.095937 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.095993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.095993 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.096232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.096050 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qw7\" (UniqueName: \"kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.096232 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:30:08.096115 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-serving-cert: secret "maas-api-e2e-shared-a-w2-54c25e-serving-cert" not found Nov 01 09:30:08.096232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.096166 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.096232 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:30:08.096200 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls podName:a63048c0-6615-43f9-909a-d79ee06fa42b nodeName:}" failed. No retries permitted until 2026-11-01 09:30:08.596182302 +0000 UTC m=+3431.684518952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" (UID: "a63048c0-6615-43f9-909a-d79ee06fa42b") : secret "maas-api-e2e-shared-a-w2-54c25e-serving-cert" not found Nov 01 09:30:08.096870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.096851 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.098838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.098812 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.154445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.154400 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qw7\" (UniqueName: \"kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.602808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.602771 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.605453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.605432 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:08.850875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:08.850835 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:09.229232 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:09.229208 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr"] Nov 01 09:30:09.231988 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:30:09.231954 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63048c0_6615_43f9_909a_d79ee06fa42b.slice/crio-c8d243897be452201d04be8d0975a17fe3a415eb3a1cd6e6dba6f510d2cc4b3c WatchSource:0}: Error finding container c8d243897be452201d04be8d0975a17fe3a415eb3a1cd6e6dba6f510d2cc4b3c: Status 404 returned error can't find the container with id c8d243897be452201d04be8d0975a17fe3a415eb3a1cd6e6dba6f510d2cc4b3c Nov 01 09:30:09.233732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:09.233704 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:30:09.266115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:09.266078 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" event={"ID":"a63048c0-6615-43f9-909a-d79ee06fa42b","Type":"ContainerStarted","Data":"c8d243897be452201d04be8d0975a17fe3a415eb3a1cd6e6dba6f510d2cc4b3c"} Nov 01 09:30:10.274577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:10.274537 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" event={"ID":"a63048c0-6615-43f9-909a-d79ee06fa42b","Type":"ContainerStarted","Data":"456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878"} Nov 01 09:30:10.275049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:10.274633 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:10.400115 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:10.400059 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" podStartSLOduration=2.465117554 podStartE2EDuration="3.400041776s" totalImagesPullingTime="934.924222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:10.39688263 +0000 UTC m=+3433.485219303" watchObservedRunningTime="2026-11-01 09:30:10.400041776 +0000 UTC m=+3433.488378448" Nov 01 09:30:14.344749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:14.344695 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv"] Nov 01 09:30:15.480822 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:30:15.480778 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:30:16.287006 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:16.286975 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:30:29.366359 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.366318 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" podUID="3d0d061f-2fcf-4ac3-882c-1bc7af451253" containerName="main" containerID="cri-o://ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89" gracePeriod=45 Nov 01 09:30:29.658961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.658929 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:30:29.760673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.760634 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs\") pod \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " Nov 01 09:30:29.760673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.760683 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home\") pod \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " Nov 01 09:30:29.760944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.760864 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir\") pod \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " Nov 01 09:30:29.760944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.760905 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk\" (UniqueName: \"kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk\") pod \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " Nov 01 09:30:29.760944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.760938 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm\") pod \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " Nov 01 09:30:29.761088 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.760982 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home" pod "3d0d061f-2fcf-4ac3-882c-1bc7af451253" (UID: "3d0d061f-2fcf-4ac3-882c-1bc7af451253"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:29.761088 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.761071 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache\") pod \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\" (UID: \"3d0d061f-2fcf-4ac3-882c-1bc7af451253\") " Nov 01 09:30:29.761200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.761132 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir" pod "3d0d061f-2fcf-4ac3-882c-1bc7af451253" (UID: "3d0d061f-2fcf-4ac3-882c-1bc7af451253"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:29.761348 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.761323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache" pod "3d0d061f-2fcf-4ac3-882c-1bc7af451253" (UID: "3d0d061f-2fcf-4ac3-882c-1bc7af451253"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:29.761619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.761601 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:29.761691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.761625 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:29.761691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.761639 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:29.763346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.763322 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm" pod "3d0d061f-2fcf-4ac3-882c-1bc7af451253" (UID: "3d0d061f-2fcf-4ac3-882c-1bc7af451253"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:30:29.763437 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.763386 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs" pod "3d0d061f-2fcf-4ac3-882c-1bc7af451253" (UID: "3d0d061f-2fcf-4ac3-882c-1bc7af451253"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:30:29.763685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.763658 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk" pod "3d0d061f-2fcf-4ac3-882c-1bc7af451253" (UID: "3d0d061f-2fcf-4ac3-882c-1bc7af451253"). InnerVolumeSpecName "kube-api-access-g8pzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:30:29.862945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.862896 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0d061f-2fcf-4ac3-882c-1bc7af451253-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:29.862945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.862936 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8pzk\" (UniqueName: \"kubernetes.io/projected/3d0d061f-2fcf-4ac3-882c-1bc7af451253-kube-api-access-g8pzk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:29.862945 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:29.862951 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0d061f-2fcf-4ac3-882c-1bc7af451253-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:30:30.486042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.486001 5710 generic.go:356] "Generic (PLEG): container finished" podID="3d0d061f-2fcf-4ac3-882c-1bc7af451253" containerID="ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89" exitCode=0 Nov 01 09:30:30.486512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.486052 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" Nov 01 09:30:30.486512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.486085 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" event={"ID":"3d0d061f-2fcf-4ac3-882c-1bc7af451253","Type":"ContainerDied","Data":"ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89"} Nov 01 09:30:30.486512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.486133 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv" event={"ID":"3d0d061f-2fcf-4ac3-882c-1bc7af451253","Type":"ContainerDied","Data":"23a585ad87960fd9cb5339311f249dbf03a4210b37a95599dd485bac4bd8e1d4"} Nov 01 09:30:30.486512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.486156 5710 scope.go:122] "RemoveContainer" containerID="ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89" Nov 01 09:30:30.498680 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.498653 5710 scope.go:122] "RemoveContainer" containerID="ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89" Nov 01 09:30:30.499069 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:30:30.499034 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89\": container with ID starting with ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89 not found: ID does not exist" containerID="ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89" Nov 01 09:30:30.499135 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.499062 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89"} err="failed to get container status \"ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89\": rpc error: code = NotFound desc = could not find container \"ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89\": container with ID starting with ca6bb62a27b0f17d030acf4ec5e0cd869efeb0bae82ce59de61b9704940e1c89 not found: ID does not exist" Nov 01 09:30:30.648378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.648336 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv"] Nov 01 09:30:30.694685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:30.694645 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fd5a10/auth-test-model-fd5a10-kserve-6cf5749857-r5ksv"] Nov 01 09:30:31.060089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:31.060057 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0d061f-2fcf-4ac3-882c-1bc7af451253" path="/var/lib/kubelet/pods/3d0d061f-2fcf-4ac3-882c-1bc7af451253/volumes" Nov 01 09:30:45.478539 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:30:45.478496 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:30:52.193984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.193944 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n"] Nov 01 09:30:52.194903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.194879 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3d0d061f-2fcf-4ac3-882c-1bc7af451253" containerName="main" Nov 01 09:30:52.194903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.194903 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0d061f-2fcf-4ac3-882c-1bc7af451253" containerName="main" Nov 01 09:30:52.195097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.195056 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0d061f-2fcf-4ac3-882c-1bc7af451253" containerName="main" Nov 01 09:30:52.200228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.200205 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.253964 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.253929 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-46bf99-openshift-default-dockercfg-2rzjn\"" Nov 01 09:30:52.298673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.298640 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n"] Nov 01 09:30:52.363986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.363953 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.363986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.363988 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364130 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364162 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364305 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364349 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrsr\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364406 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364372 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364429 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.364519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.364506 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465214 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrsr\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465257 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465281 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465340 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465411 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465613 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465433 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465682 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465819 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465766 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465900 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465684 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.465991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.465972 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.466216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.466183 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.466216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.466208 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.466390 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.466302 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.468638 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.468611 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.468837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.468819 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.523085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.523049 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.523284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.523128 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrsr\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:52.814048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:52.813951 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:53.245608 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.245580 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n"] Nov 01 09:30:53.248021 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:30:53.247993 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda349c365_47f8_447e_aed4_288a8d4b67b9.slice/crio-adff07d02f89f999e25de7922def4dba75e4249f19b8fc62cb15c7b1bc135fd4 WatchSource:0}: Error finding container adff07d02f89f999e25de7922def4dba75e4249f19b8fc62cb15c7b1bc135fd4: Status 404 returned error can't find the container with id adff07d02f89f999e25de7922def4dba75e4249f19b8fc62cb15c7b1bc135fd4 Nov 01 09:30:53.250786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.250753 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:30:53.250867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.250823 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:30:53.250867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.250853 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:30:53.661245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.661204 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" event={"ID":"a349c365-47f8-447e-aed4-288a8d4b67b9","Type":"ContainerStarted","Data":"85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b"} Nov 01 09:30:53.661446 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.661282 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" event={"ID":"a349c365-47f8-447e-aed4-288a8d4b67b9","Type":"ContainerStarted","Data":"adff07d02f89f999e25de7922def4dba75e4249f19b8fc62cb15c7b1bc135fd4"} Nov 01 09:30:53.774587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.774513 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podStartSLOduration=1.774492779 podStartE2EDuration="1.774492779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:30:53.774097948 +0000 UTC m=+3476.862434619" watchObservedRunningTime="2026-11-01 09:30:53.774492779 +0000 UTC m=+3476.862829454" Nov 01 09:30:53.814444 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:53.814407 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:54.816665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:54.816632 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 09:30:54.817210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:54.816690 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 09:30:55.815022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:55.814978 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 09:30:55.815220 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:55.815059 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 09:30:56.815426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:56.815377 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 09:30:56.815884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:56.815450 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 09:30:57.814875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:57.814832 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 09:30:57.815076 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:57.814899 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 09:30:58.814396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:58.814341 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 01 09:30:58.814948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:58.814424 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 01 09:30:59.991905 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:59.991865 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:59.993239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:59.993206 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:30:59.994842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:30:59.994817 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:31:04.968776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:04.968736 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c"] Nov 01 09:31:04.974785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:04.974756 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.024114 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.024081 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e3728d-openshift-default-dockercfg-kcbx9\"" Nov 01 09:31:05.073436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.073402 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c"] Nov 01 09:31:05.099212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099161 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099231 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099386 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099537 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475vp\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099580 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099607 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099662 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.099756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099704 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.100016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.099818 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.201170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.201125 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.201483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.201451 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.201629 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.201502 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.201629 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.201559 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.201804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.201641 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.201804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.201660 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.202151 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.202125 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.202801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.202384 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.202958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.202414 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.203130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.202698 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.203297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.202917 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.203449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.203114 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.205085 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.204421 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.205360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.205340 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475vp\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.209741 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.209644 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.247439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.247348 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.281916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.281874 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.285756 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.285648 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-475vp\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp\") pod \"e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.292822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.291050 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:05.771957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.771927 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c"] Nov 01 09:31:05.773555 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:31:05.773522 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea276ab2_0e20_4741_a42c_ccf8165de1cd.slice/crio-c58a13318938d7ecc452cfa704e34eb2ba1ababb7acfc6dccca23c842ad8926d WatchSource:0}: Error finding container c58a13318938d7ecc452cfa704e34eb2ba1ababb7acfc6dccca23c842ad8926d: Status 404 returned error can't find the container with id c58a13318938d7ecc452cfa704e34eb2ba1ababb7acfc6dccca23c842ad8926d Nov 01 09:31:05.776196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.776160 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:31:05.776285 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.776236 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:31:05.776285 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:05.776265 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:31:06.786505 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:06.786461 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" event={"ID":"ea276ab2-0e20-4741-a42c-ccf8165de1cd","Type":"ContainerStarted","Data":"3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba"} Nov 01 09:31:06.786977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:06.786530 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" event={"ID":"ea276ab2-0e20-4741-a42c-ccf8165de1cd","Type":"ContainerStarted","Data":"c58a13318938d7ecc452cfa704e34eb2ba1ababb7acfc6dccca23c842ad8926d"} Nov 01 09:31:06.947491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:06.947424 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podStartSLOduration=2.9474072529999997 podStartE2EDuration="2.947407253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:31:06.94518025 +0000 UTC m=+3490.033516928" watchObservedRunningTime="2026-11-01 09:31:06.947407253 +0000 UTC m=+3490.035743925" Nov 01 09:31:07.293735 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:07.293680 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:07.300779 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:07.300747 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:07.800314 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:07.800283 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:07.801282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:07.801256 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:31:12.355158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:12.355118 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:31:15.481160 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:31:15.481112 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:31:33.026037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:33.025995 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd"] Nov 01 09:31:33.028615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:33.026212 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" containerID="cri-o://703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1" gracePeriod=30 Nov 01 09:31:38.301917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.301843 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:31:38.354271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354230 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354312 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354349 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354397 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354435 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354466 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354582 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354651 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354664 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:31:38.354846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354754 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy\") pod \"182f210f-0aa2-44bb-b7c7-84d04871716e\" (UID: \"182f210f-0aa2-44bb-b7c7-84d04871716e\") " Nov 01 09:31:38.354846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354807 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:31:38.354846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354820 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:31:38.355330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354960 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:31:38.355330 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.354987 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:31:38.355460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.355426 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.355460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.355445 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182f210f-0aa2-44bb-b7c7-84d04871716e-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.355460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.355454 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.355636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.355463 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.355636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.355472 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.357069 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.357039 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:31:38.357203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.357116 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:31:38.357203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.357157 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "kube-api-access-xbx8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:31:38.357386 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.357368 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo" pod "182f210f-0aa2-44bb-b7c7-84d04871716e" (UID: "182f210f-0aa2-44bb-b7c7-84d04871716e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:31:38.456243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.456203 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbx8h\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-kube-api-access-xbx8h\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.456243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.456241 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.456475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.456252 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:38.456475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:38.456263 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182f210f-0aa2-44bb-b7c7-84d04871716e-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:31:39.029612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.029561 5710 generic.go:356] "Generic (PLEG): container finished" podID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerID="703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1" exitCode=0 Nov 01 09:31:39.029810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.029626 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" Nov 01 09:31:39.029810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.029635 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" event={"ID":"182f210f-0aa2-44bb-b7c7-84d04871716e","Type":"ContainerDied","Data":"703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1"} Nov 01 09:31:39.029810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.029692 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd" event={"ID":"182f210f-0aa2-44bb-b7c7-84d04871716e","Type":"ContainerDied","Data":"a80644a64d41d531c39aea1939ee3440c57969b6b98b67f9f664260e7de62609"} Nov 01 09:31:39.029810 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.029754 5710 scope.go:122] "RemoveContainer" containerID="703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1" Nov 01 09:31:39.042464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.042442 5710 scope.go:122] "RemoveContainer" containerID="e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9" Nov 01 09:31:39.053164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.053142 5710 scope.go:122] "RemoveContainer" containerID="703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1" Nov 01 09:31:39.053527 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:31:39.053496 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1\": container with ID starting with 703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1 not found: ID does not exist" containerID="703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1" Nov 01 09:31:39.053619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.053530 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1"} err="failed to get container status \"703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1\": rpc error: code = NotFound desc = could not find container \"703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1\": container with ID starting with 703a7a95d60888c37dfec1af08f7c7f37b254786cdfe6dc343ee229c0a045ff1 not found: ID does not exist" Nov 01 09:31:39.053619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.053560 5710 scope.go:122] "RemoveContainer" containerID="e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9" Nov 01 09:31:39.053847 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:31:39.053826 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9\": container with ID starting with e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9 not found: ID does not exist" containerID="e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9" Nov 01 09:31:39.053902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.053855 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9"} err="failed to get container status \"e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9\": rpc error: code = NotFound desc = could not find container \"e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9\": container with ID starting with e16fcf2c5e65a9dd4ba619429b9ade34bf57a8160993461fa84d0bee96a27ec9 not found: ID does not exist" Nov 01 09:31:39.231980 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.231943 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd"] Nov 01 09:31:39.278623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:39.278581 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a83714-openshift-default-7cf98fb789-5chkd"] Nov 01 09:31:41.059291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:41.059257 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" path="/var/lib/kubelet/pods/182f210f-0aa2-44bb-b7c7-84d04871716e/volumes" Nov 01 09:31:45.478638 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:31:45.478598 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:31:48.507148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:31:48.507105 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:32:15.477928 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:32:15.477892 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:32:29.190772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:29.190712 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74"] Nov 01 09:32:29.191365 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:29.190951 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" containerID="cri-o://f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc" gracePeriod=30 Nov 01 09:32:29.351319 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:29.351272 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:32:30.283185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:30.283137 5710 patch_prober.go:29] interesting pod/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:32:30.283606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:30.283217 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:32:34.473169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.473102 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:32:34.539668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539633 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.539668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539669 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.539959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539739 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.539959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539781 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.539959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539835 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.539959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539944 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.540198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539986 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.540198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.539987 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:32:34.540198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540142 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.540198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540156 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:32:34.540198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540171 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:32:34.540198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540186 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy\") pod \"6bea5470-78e2-4de4-966a-548c85083527\" (UID: \"6bea5470-78e2-4de4-966a-548c85083527\") " Nov 01 09:32:34.540507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540229 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:32:34.540507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540399 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:32:34.540801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540684 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.540801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540708 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bea5470-78e2-4de4-966a-548c85083527-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.540801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540749 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.540801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540765 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.540801 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.540779 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.542335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.542316 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:32:34.542426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.542407 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "kube-api-access-qvhws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:32:34.542791 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.542769 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:32:34.542883 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.542793 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy" pod "6bea5470-78e2-4de4-966a-548c85083527" (UID: "6bea5470-78e2-4de4-966a-548c85083527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:32:34.566838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.566808 5710 generic.go:356] "Generic (PLEG): container finished" podID="6bea5470-78e2-4de4-966a-548c85083527" containerID="f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc" exitCode=0 Nov 01 09:32:34.566942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.566861 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" Nov 01 09:32:34.566942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.566893 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" event={"ID":"6bea5470-78e2-4de4-966a-548c85083527","Type":"ContainerDied","Data":"f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc"} Nov 01 09:32:34.567021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.566951 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74" event={"ID":"6bea5470-78e2-4de4-966a-548c85083527","Type":"ContainerDied","Data":"22163fc254a296684862c53d4ef7f557e40f310076b1326adb91e8fd02281112"} Nov 01 09:32:34.567021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.566979 5710 scope.go:122] "RemoveContainer" containerID="f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc" Nov 01 09:32:34.578181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.578166 5710 scope.go:122] "RemoveContainer" containerID="f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc" Nov 01 09:32:34.578468 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:32:34.578447 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc\": container with ID starting with f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc not found: ID does not exist" containerID="f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc" Nov 01 09:32:34.578537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.578478 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc"} err="failed to get container status \"f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc\": rpc error: code = NotFound desc = could not find container \"f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc\": container with ID starting with f5fc221d4a8a13482cb88900243935d87ef1ffe66af0203d6881b3840e37e4cc not found: ID does not exist" Nov 01 09:32:34.642560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.642527 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bea5470-78e2-4de4-966a-548c85083527-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.642560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.642559 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bea5470-78e2-4de4-966a-548c85083527-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.642831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.642574 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhws\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-kube-api-access-qvhws\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.642831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.642586 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bea5470-78e2-4de4-966a-548c85083527-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:32:34.743355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.743248 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74"] Nov 01 09:32:34.749376 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:34.749334 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-54e01b-openshift-default-5dd6546849-86r74"] Nov 01 09:32:35.059751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:32:35.059619 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bea5470-78e2-4de4-966a-548c85083527" path="/var/lib/kubelet/pods/6bea5470-78e2-4de4-966a-548c85083527/volumes" Nov 01 09:32:45.477665 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:32:45.477628 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:33:01.144655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:01.144618 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:33:01.159515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:01.159486 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:33:02.138812 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:02.138757 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.191485 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w"] Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.192495 5710 cpu_manager.go:414] "containerMap: removing container" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.192517 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.192576 5710 cpu_manager.go:414] "containerMap: removing container" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.192585 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.192595 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" Nov 01 09:33:05.198809 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.192602 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" Nov 01 09:33:05.205954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.205920 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" Nov 01 09:33:05.206146 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.205972 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bea5470-78e2-4de4-966a-548c85083527" containerName="istio-proxy" Nov 01 09:33:05.211384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.211361 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.242177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.242145 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w"] Nov 01 09:33:05.264683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.264642 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.264885 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.264697 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.264885 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.264856 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.264979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.264947 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhtd\" (UniqueName: \"kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.305834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.305784 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f8071b-serving-cert\"" Nov 01 09:33:05.366633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.366597 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.366633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.366638 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.366972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.366682 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhtd\" (UniqueName: \"kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.366972 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:33:05.366783 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-serving-cert: secret "maas-api-e2e-shared-b-w2-f8071b-serving-cert" not found Nov 01 09:33:05.366972 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:33:05.366851 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls podName:210ec782-1646-4570-a242-f458eec18a3b nodeName:}" failed. No retries permitted until 2026-11-01 09:33:05.86683589 +0000 UTC m=+3608.955172543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" (UID: "210ec782-1646-4570-a242-f458eec18a3b") : secret "maas-api-e2e-shared-b-w2-f8071b-serving-cert" not found Nov 01 09:33:05.366972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.366876 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.367530 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.367510 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.369583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.369562 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.453045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.449866 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhtd\" (UniqueName: \"kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.877680 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.877631 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:05.880397 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:05.880374 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:06.125803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:06.125761 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:06.529317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:06.529223 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w"] Nov 01 09:33:06.537746 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:33:06.535565 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210ec782_1646_4570_a242_f458eec18a3b.slice/crio-1028bd3fee953b77d509591d066c9285fbafb479f4d1739937de78bf937b3502 WatchSource:0}: Error finding container 1028bd3fee953b77d509591d066c9285fbafb479f4d1739937de78bf937b3502: Status 404 returned error can't find the container with id 1028bd3fee953b77d509591d066c9285fbafb479f4d1739937de78bf937b3502 Nov 01 09:33:06.826373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:06.826279 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" event={"ID":"210ec782-1646-4570-a242-f458eec18a3b","Type":"ContainerStarted","Data":"1028bd3fee953b77d509591d066c9285fbafb479f4d1739937de78bf937b3502"} Nov 01 09:33:07.835543 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:07.835433 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" event={"ID":"210ec782-1646-4570-a242-f458eec18a3b","Type":"ContainerStarted","Data":"a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4"} Nov 01 09:33:07.835543 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:07.835501 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:07.962169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:07.962099 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" podStartSLOduration=1.972214064 podStartE2EDuration="2.962080711s" totalImagesPullingTime="989.866647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:07.958981708 +0000 UTC m=+3611.047318382" watchObservedRunningTime="2026-11-01 09:33:07.962080711 +0000 UTC m=+3611.050417383" Nov 01 09:33:13.848043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.848007 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:33:13.878938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.878903 5710 generic.go:356] "Generic (PLEG): container finished" podID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerID="3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba" exitCode=0 Nov 01 09:33:13.879143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.878974 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" event={"ID":"ea276ab2-0e20-4741-a42c-ccf8165de1cd","Type":"ContainerDied","Data":"3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba"} Nov 01 09:33:13.879442 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.879427 5710 scope.go:122] "RemoveContainer" containerID="3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba" Nov 01 09:33:13.881297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.881263 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:33:13.881411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.881339 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:33:13.881411 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:13.881371 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:33:14.887225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:14.887187 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" event={"ID":"ea276ab2-0e20-4741-a42c-ccf8165de1cd","Type":"ContainerStarted","Data":"829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4"} Nov 01 09:33:15.294101 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:15.294066 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:33:15.294541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:15.294511 5710 patch_prober.go:29] interesting pod/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 09:33:15.294747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:15.294701 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 09:33:15.499745 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:33:15.496941 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:33:16.297747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:16.296952 5710 patch_prober.go:29] interesting pod/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 09:33:16.297747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:16.297013 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 09:33:17.298244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:17.298199 5710 patch_prober.go:29] interesting pod/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 09:33:17.298948 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:17.298910 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 09:33:18.295467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:18.295424 5710 patch_prober.go:29] interesting pod/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 09:33:18.295850 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:18.295811 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 09:33:19.301747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:19.301282 5710 patch_prober.go:29] interesting pod/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 09:33:19.301747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:19.301365 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 09:33:20.298745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:20.297145 5710 patch_prober.go:29] interesting pod/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 09:33:20.298745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:20.297215 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 09:33:21.298327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:21.298283 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:33:21.298782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:21.298675 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:33:21.299358 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:21.299341 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:33:25.876162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.876124 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n"] Nov 01 09:33:25.877082 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.877062 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="182f210f-0aa2-44bb-b7c7-84d04871716e" containerName="istio-proxy" Nov 01 09:33:25.880926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.880908 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913264 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913448 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913297 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913448 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913413 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913447 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgk8x\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913483 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913606 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913680 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913742 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.913843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.913786 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:25.930060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.930021 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b0d0f3-openshift-default-dockercfg-qc7fc\"" Nov 01 09:33:25.978993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:25.978955 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n"] Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.036401 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038349 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038468 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038506 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk8x\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038553 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038619 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038660 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038696 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.038782 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.040017 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.043753 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.040325 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.045072 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.045046 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.045303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.045275 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.045829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.045799 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.046431 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.046409 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.049754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.049112 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.096424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.096397 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.102131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.102091 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk8x\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x\") pod \"e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.194711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.194634 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:26.656321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.656283 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n"] Nov 01 09:33:26.661059 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:33:26.661026 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1614416_7c0a_4515_bbcc_1be46726f747.slice/crio-86e69dfc3d5f602712c67b6bf15e3d491a4a17422f314317a6989b91886747bd WatchSource:0}: Error finding container 86e69dfc3d5f602712c67b6bf15e3d491a4a17422f314317a6989b91886747bd: Status 404 returned error can't find the container with id 86e69dfc3d5f602712c67b6bf15e3d491a4a17422f314317a6989b91886747bd Nov 01 09:33:26.663864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.663828 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:33:26.663966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.663928 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:33:26.663966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.663959 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:33:26.986382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.986280 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" event={"ID":"e1614416-7c0a-4515-bbcc-1be46726f747","Type":"ContainerStarted","Data":"708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa"} Nov 01 09:33:26.986382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:26.986347 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" event={"ID":"e1614416-7c0a-4515-bbcc-1be46726f747","Type":"ContainerStarted","Data":"86e69dfc3d5f602712c67b6bf15e3d491a4a17422f314317a6989b91886747bd"} Nov 01 09:33:27.102542 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:27.102466 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podStartSLOduration=2.102447049 podStartE2EDuration="2.102447049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:27.099156677 +0000 UTC m=+3630.187493344" watchObservedRunningTime="2026-11-01 09:33:27.102447049 +0000 UTC m=+3630.190783724" Nov 01 09:33:27.195811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:27.195771 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:27.202219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:27.202189 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:27.993153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:27.993118 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:27.994335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:27.994313 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:33:43.387986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:43.387937 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:33:45.480517 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:33:45.480472 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:33:46.902143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:46.902109 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht"] Nov 01 09:33:46.906467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:46.906445 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:46.948834 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:46.948782 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlstc\" (UniqueName: \"kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc\") pod \"payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:46.949038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:46.948897 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:46.960092 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:46.960064 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht"] Nov 01 09:33:47.013747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.006949 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1f3121\"" Nov 01 09:33:47.013747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.007046 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1f3121-dockercfg-bd2hj\"" Nov 01 09:33:47.050962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.050923 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlstc\" (UniqueName: \"kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc\") pod \"payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:47.051299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.051271 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:47.053814 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.052662 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:47.084808 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.084774 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd"] Nov 01 09:33:47.089601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.089575 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.116308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.116273 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd"] Nov 01 09:33:47.116792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.116764 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlstc\" (UniqueName: \"kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc\") pod \"payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:47.153244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.153141 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume\") pod \"payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.153453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.153398 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6jg\" (UniqueName: \"kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg\") pod \"payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.221561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.221525 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:47.255117 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.255070 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6jg\" (UniqueName: \"kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg\") pod \"payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.255304 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.255261 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume\") pod \"payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.256226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.256197 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume\") pod \"payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.313467 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.313437 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6jg\" (UniqueName: \"kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg\") pod \"payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.404506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.404422 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:47.600807 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.600780 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht"] Nov 01 09:33:47.602768 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:33:47.602708 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee838c44_49b7_4a23_8268_bf09f6d00205.slice/crio-e1ccf21f62a8bb7bfc32de36773e313b07e0593b055d66beaa2e1c8b77bbc1f7 WatchSource:0}: Error finding container e1ccf21f62a8bb7bfc32de36773e313b07e0593b055d66beaa2e1c8b77bbc1f7: Status 404 returned error can't find the container with id e1ccf21f62a8bb7bfc32de36773e313b07e0593b055d66beaa2e1c8b77bbc1f7 Nov 01 09:33:47.788116 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:47.788084 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd"] Nov 01 09:33:47.788673 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:33:47.788648 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439e34dd_4059_4223_b2ba_52a4e44e0fd9.slice/crio-e9bb7d7b48054ecc6f69ec98caf0a1931967ca2a5e0eafff1ea264f15bd1039a WatchSource:0}: Error finding container e9bb7d7b48054ecc6f69ec98caf0a1931967ca2a5e0eafff1ea264f15bd1039a: Status 404 returned error can't find the container with id e9bb7d7b48054ecc6f69ec98caf0a1931967ca2a5e0eafff1ea264f15bd1039a Nov 01 09:33:48.145453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.145416 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" event={"ID":"439e34dd-4059-4223-b2ba-52a4e44e0fd9","Type":"ContainerStarted","Data":"944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564"} Nov 01 09:33:48.145935 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.145473 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" event={"ID":"439e34dd-4059-4223-b2ba-52a4e44e0fd9","Type":"ContainerStarted","Data":"e9bb7d7b48054ecc6f69ec98caf0a1931967ca2a5e0eafff1ea264f15bd1039a"} Nov 01 09:33:48.145935 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.145499 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:33:48.147267 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.147236 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" event={"ID":"ee838c44-49b7-4a23-8268-bf09f6d00205","Type":"ContainerStarted","Data":"33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2"} Nov 01 09:33:48.147394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.147291 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:48.147394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.147308 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" event={"ID":"ee838c44-49b7-4a23-8268-bf09f6d00205","Type":"ContainerStarted","Data":"e1ccf21f62a8bb7bfc32de36773e313b07e0593b055d66beaa2e1c8b77bbc1f7"} Nov 01 09:33:48.259395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.259336 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" podStartSLOduration=1.259319149 podStartE2EDuration="1.259319149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:48.257455742 +0000 UTC m=+3651.345792414" watchObservedRunningTime="2026-11-01 09:33:48.259319149 +0000 UTC m=+3651.347655818" Nov 01 09:33:48.382755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:48.380187 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" podStartSLOduration=2.380143482 podStartE2EDuration="2.380143482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:33:48.3671897 +0000 UTC m=+3651.455526379" watchObservedRunningTime="2026-11-01 09:33:48.380143482 +0000 UTC m=+3651.468480171" Nov 01 09:33:59.157434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:59.157391 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:33:59.157962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:33:59.157476 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:34:15.482164 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:34:15.482120 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:34:33.312577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:34:33.312538 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:34:45.508165 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:34:45.508080 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:35:15.478807 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:35:15.478767 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:35:25.257831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:25.257788 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:35:35.886364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:35.886328 5710 generic.go:356] "Generic (PLEG): container finished" podID="e1614416-7c0a-4515-bbcc-1be46726f747" containerID="708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa" exitCode=0 Nov 01 09:35:35.886847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:35.886398 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" event={"ID":"e1614416-7c0a-4515-bbcc-1be46726f747","Type":"ContainerDied","Data":"708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa"} Nov 01 09:35:35.886847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:35.886843 5710 scope.go:122] "RemoveContainer" containerID="708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa" Nov 01 09:35:35.888590 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:35.888564 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:35:35.888650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:35.888634 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:35:35.888683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:35.888667 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:35:36.914805 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:36.914707 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" event={"ID":"e1614416-7c0a-4515-bbcc-1be46726f747","Type":"ContainerStarted","Data":"7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0"} Nov 01 09:35:37.198295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:37.198191 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:35:37.198704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:37.198591 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 09:35:37.198933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:37.198907 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 09:35:38.200743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:38.200209 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 09:35:38.200743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:38.200295 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 09:35:39.196294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:39.196245 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 09:35:39.196615 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:39.196581 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 09:35:40.206318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:40.206215 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 09:35:40.206318 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:40.206292 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 09:35:41.198195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:41.198152 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 09:35:41.198995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:41.198520 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 09:35:42.218117 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:42.218078 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:35:42.219048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:42.219018 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:35:42.219937 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:42.219914 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:35:45.480913 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:35:45.480871 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:35:58.011316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.011281 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr"] Nov 01 09:35:58.011908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.011480 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" podUID="a63048c0-6615-43f9-909a-d79ee06fa42b" containerName="maas-api" containerID="cri-o://456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878" gracePeriod=30 Nov 01 09:35:58.322837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.322806 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:35:58.490862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.490829 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca\") pod \"a63048c0-6615-43f9-909a-d79ee06fa42b\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " Nov 01 09:35:58.491073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.490923 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs\") pod \"a63048c0-6615-43f9-909a-d79ee06fa42b\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " Nov 01 09:35:58.491073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.490974 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\") pod \"a63048c0-6615-43f9-909a-d79ee06fa42b\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " Nov 01 09:35:58.491173 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.491077 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7\" (UniqueName: \"kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7\") pod \"a63048c0-6615-43f9-909a-d79ee06fa42b\" (UID: \"a63048c0-6615-43f9-909a-d79ee06fa42b\") " Nov 01 09:35:58.491299 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.491186 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca" pod "a63048c0-6615-43f9-909a-d79ee06fa42b" (UID: "a63048c0-6615-43f9-909a-d79ee06fa42b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:35:58.491627 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.491604 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a63048c0-6615-43f9-909a-d79ee06fa42b-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:35:58.493456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.493433 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls" pod "a63048c0-6615-43f9-909a-d79ee06fa42b" (UID: "a63048c0-6615-43f9-909a-d79ee06fa42b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:35:58.493578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.493472 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs" pod "a63048c0-6615-43f9-909a-d79ee06fa42b" (UID: "a63048c0-6615-43f9-909a-d79ee06fa42b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:35:58.493578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.493515 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7" pod "a63048c0-6615-43f9-909a-d79ee06fa42b" (UID: "a63048c0-6615-43f9-909a-d79ee06fa42b"). InnerVolumeSpecName "kube-api-access-n9qw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:35:58.593396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.593365 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:35:58.593396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.593396 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a63048c0-6615-43f9-909a-d79ee06fa42b-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:35:58.593605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:58.593407 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9qw7\" (UniqueName: \"kubernetes.io/projected/a63048c0-6615-43f9-909a-d79ee06fa42b-kube-api-access-n9qw7\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:35:59.062381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.062284 5710 generic.go:356] "Generic (PLEG): container finished" podID="a63048c0-6615-43f9-909a-d79ee06fa42b" containerID="456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878" exitCode=0 Nov 01 09:35:59.062381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.062331 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" event={"ID":"a63048c0-6615-43f9-909a-d79ee06fa42b","Type":"ContainerDied","Data":"456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878"} Nov 01 09:35:59.062381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.062358 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" Nov 01 09:35:59.062981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.062400 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr" event={"ID":"a63048c0-6615-43f9-909a-d79ee06fa42b","Type":"ContainerDied","Data":"c8d243897be452201d04be8d0975a17fe3a415eb3a1cd6e6dba6f510d2cc4b3c"} Nov 01 09:35:59.062981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.062432 5710 scope.go:122] "RemoveContainer" containerID="456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878" Nov 01 09:35:59.091291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.091265 5710 scope.go:122] "RemoveContainer" containerID="456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878" Nov 01 09:35:59.092041 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:35:59.092004 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878\": container with ID starting with 456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878 not found: ID does not exist" containerID="456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878" Nov 01 09:35:59.092191 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.092065 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878"} err="failed to get container status \"456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878\": rpc error: code = NotFound desc = could not find container \"456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878\": container with ID starting with 456e5fb3e6eadbd8083645484211d048adc677e9f8ebb1d2c69be6f405342878 not found: ID does not exist" Nov 01 09:35:59.223385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.223350 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr"] Nov 01 09:35:59.269785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:35:59.269745 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-54c25e-675cc84cc6-lbtrr"] Nov 01 09:36:01.060344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:01.060308 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63048c0-6615-43f9-909a-d79ee06fa42b" path="/var/lib/kubelet/pods/a63048c0-6615-43f9-909a-d79ee06fa42b/volumes" Nov 01 09:36:04.303626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:04.303534 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:36:09.562327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:09.562288 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n"] Nov 01 09:36:09.562804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:09.562538 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" containerID="cri-o://85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b" gracePeriod=30 Nov 01 09:36:14.847526 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.847495 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:36:14.944802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.944711 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945026 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.944854 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945026 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.944910 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945026 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.944968 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945205 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945101 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945212 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945268 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945262 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945280 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:36:14.945375 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945331 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945382 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket\") pod \"a349c365-47f8-447e-aed4-288a8d4b67b9\" (UID: \"a349c365-47f8-447e-aed4-288a8d4b67b9\") " Nov 01 09:36:14.945476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945401 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:36:14.945476 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945438 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:36:14.945693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945606 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:36:14.945898 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.945810 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:36:14.946030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.946009 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:14.946098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.946036 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:14.946098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.946051 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:14.946098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.946064 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a349c365-47f8-447e-aed4-288a8d4b67b9-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:14.946098 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.946078 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:14.947523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.947500 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:36:14.947523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.947504 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:36:14.947663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.947525 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "kube-api-access-dvrsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:36:14.947745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:14.947711 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo" pod "a349c365-47f8-447e-aed4-288a8d4b67b9" (UID: "a349c365-47f8-447e-aed4-288a8d4b67b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:36:15.047340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.047305 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:15.047340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.047338 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrsr\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-kube-api-access-dvrsr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:15.047340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.047350 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:15.047595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.047358 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a349c365-47f8-447e-aed4-288a8d4b67b9-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:15.204329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.204288 5710 generic.go:356] "Generic (PLEG): container finished" podID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerID="85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b" exitCode=0 Nov 01 09:36:15.204549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.204358 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" Nov 01 09:36:15.204549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.204387 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" event={"ID":"a349c365-47f8-447e-aed4-288a8d4b67b9","Type":"ContainerDied","Data":"85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b"} Nov 01 09:36:15.204549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.204449 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n" event={"ID":"a349c365-47f8-447e-aed4-288a8d4b67b9","Type":"ContainerDied","Data":"adff07d02f89f999e25de7922def4dba75e4249f19b8fc62cb15c7b1bc135fd4"} Nov 01 09:36:15.204549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.204474 5710 scope.go:122] "RemoveContainer" containerID="85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b" Nov 01 09:36:15.215280 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.215259 5710 scope.go:122] "RemoveContainer" containerID="85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b" Nov 01 09:36:15.215639 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:36:15.215618 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b\": container with ID starting with 85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b not found: ID does not exist" containerID="85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b" Nov 01 09:36:15.215703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.215651 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b"} err="failed to get container status \"85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b\": rpc error: code = NotFound desc = could not find container \"85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b\": container with ID starting with 85f4f5e2056be593ac5bc8a4752efbf7eaab1aecd4699a7be48b131be2e3bf7b not found: ID does not exist" Nov 01 09:36:15.370481 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.370439 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n"] Nov 01 09:36:15.373520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:15.373488 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-h4n4n"] Nov 01 09:36:15.477653 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:36:15.477576 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:36:17.058857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:17.058822 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" path="/var/lib/kubelet/pods/a349c365-47f8-447e-aed4-288a8d4b67b9/volumes" Nov 01 09:36:39.150044 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:39.150001 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:36:45.479086 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:36:45.479036 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:36:53.390565 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:53.390511 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd"] Nov 01 09:36:53.489598 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:53.489564 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht"] Nov 01 09:36:58.391238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.391181 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" podUID="439e34dd-4059-4223-b2ba-52a4e44e0fd9" containerName="payload-processing" containerID="cri-o://944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564" gracePeriod=25 Nov 01 09:36:58.490229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.490187 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" podUID="ee838c44-49b7-4a23-8268-bf09f6d00205" containerName="payload-pre-processing" containerID="cri-o://33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2" gracePeriod=25 Nov 01 09:36:58.677263 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.677235 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:36:58.695908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.695871 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume\") pod \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " Nov 01 09:36:58.697213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.696371 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg\" (UniqueName: \"kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg\") pod \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\" (UID: \"439e34dd-4059-4223-b2ba-52a4e44e0fd9\") " Nov 01 09:36:58.697213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.696658 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume" pod "439e34dd-4059-4223-b2ba-52a4e44e0fd9" (UID: "439e34dd-4059-4223-b2ba-52a4e44e0fd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:36:58.697213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.697168 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/439e34dd-4059-4223-b2ba-52a4e44e0fd9-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:58.703311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.703276 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg" pod "439e34dd-4059-4223-b2ba-52a4e44e0fd9" (UID: "439e34dd-4059-4223-b2ba-52a4e44e0fd9"). InnerVolumeSpecName "kube-api-access-9q6jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:36:58.770584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.770558 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:36:58.797975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.797936 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc\" (UniqueName: \"kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc\") pod \"ee838c44-49b7-4a23-8268-bf09f6d00205\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " Nov 01 09:36:58.798178 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.798107 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume\") pod \"ee838c44-49b7-4a23-8268-bf09f6d00205\" (UID: \"ee838c44-49b7-4a23-8268-bf09f6d00205\") " Nov 01 09:36:58.798620 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.798599 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6jg\" (UniqueName: \"kubernetes.io/projected/439e34dd-4059-4223-b2ba-52a4e44e0fd9-kube-api-access-9q6jg\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:58.798960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.798940 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume" pod "ee838c44-49b7-4a23-8268-bf09f6d00205" (UID: "ee838c44-49b7-4a23-8268-bf09f6d00205"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:36:58.801378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.801346 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc" pod "ee838c44-49b7-4a23-8268-bf09f6d00205" (UID: "ee838c44-49b7-4a23-8268-bf09f6d00205"). InnerVolumeSpecName "kube-api-access-vlstc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:36:58.899410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.899323 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee838c44-49b7-4a23-8268-bf09f6d00205-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:58.899410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:58.899354 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlstc\" (UniqueName: \"kubernetes.io/projected/ee838c44-49b7-4a23-8268-bf09f6d00205-kube-api-access-vlstc\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:36:59.490678 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.490644 5710 generic.go:356] "Generic (PLEG): container finished" podID="ee838c44-49b7-4a23-8268-bf09f6d00205" containerID="33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2" exitCode=0 Nov 01 09:36:59.491184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.490705 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" Nov 01 09:36:59.491184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.490733 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" event={"ID":"ee838c44-49b7-4a23-8268-bf09f6d00205","Type":"ContainerDied","Data":"33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2"} Nov 01 09:36:59.491184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.490787 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht" event={"ID":"ee838c44-49b7-4a23-8268-bf09f6d00205","Type":"ContainerDied","Data":"e1ccf21f62a8bb7bfc32de36773e313b07e0593b055d66beaa2e1c8b77bbc1f7"} Nov 01 09:36:59.491184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.490816 5710 scope.go:122] "RemoveContainer" containerID="33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2" Nov 01 09:36:59.492455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.492431 5710 generic.go:356] "Generic (PLEG): container finished" podID="439e34dd-4059-4223-b2ba-52a4e44e0fd9" containerID="944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564" exitCode=0 Nov 01 09:36:59.492540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.492497 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" event={"ID":"439e34dd-4059-4223-b2ba-52a4e44e0fd9","Type":"ContainerDied","Data":"944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564"} Nov 01 09:36:59.492540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.492504 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" Nov 01 09:36:59.492540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.492535 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd" event={"ID":"439e34dd-4059-4223-b2ba-52a4e44e0fd9","Type":"ContainerDied","Data":"e9bb7d7b48054ecc6f69ec98caf0a1931967ca2a5e0eafff1ea264f15bd1039a"} Nov 01 09:36:59.501520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.501498 5710 scope.go:122] "RemoveContainer" containerID="33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2" Nov 01 09:36:59.501917 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:36:59.501891 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2\": container with ID starting with 33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2 not found: ID does not exist" containerID="33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2" Nov 01 09:36:59.502043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.501922 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2"} err="failed to get container status \"33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2\": rpc error: code = NotFound desc = could not find container \"33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2\": container with ID starting with 33441ade703e064dff4c9d162e73f3c1f69bffcef04b26e482c4f92940573af2 not found: ID does not exist" Nov 01 09:36:59.502043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.501943 5710 scope.go:122] "RemoveContainer" containerID="944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564" Nov 01 09:36:59.512663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.512636 5710 scope.go:122] "RemoveContainer" containerID="944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564" Nov 01 09:36:59.513072 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:36:59.513043 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564\": container with ID starting with 944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564 not found: ID does not exist" containerID="944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564" Nov 01 09:36:59.513159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.513085 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564"} err="failed to get container status \"944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564\": rpc error: code = NotFound desc = could not find container \"944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564\": container with ID starting with 944826c19f2696ddd9a29cc4b33598a102a17dfb07c573ae03651a67b9a59564 not found: ID does not exist" Nov 01 09:36:59.647491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.647454 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht"] Nov 01 09:36:59.697217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.697127 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1f3121-7f678869cf-vbsht"] Nov 01 09:36:59.801143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.801105 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd"] Nov 01 09:36:59.804864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:36:59.804837 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1f3121-55dddfbbfc-tx7dd"] Nov 01 09:37:01.060541 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:01.060506 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439e34dd-4059-4223-b2ba-52a4e44e0fd9" path="/var/lib/kubelet/pods/439e34dd-4059-4223-b2ba-52a4e44e0fd9/volumes" Nov 01 09:37:01.060991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:01.060946 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee838c44-49b7-4a23-8268-bf09f6d00205" path="/var/lib/kubelet/pods/ee838c44-49b7-4a23-8268-bf09f6d00205/volumes" Nov 01 09:37:15.479084 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:37:15.479041 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:37:23.159316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:23.159272 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c"] Nov 01 09:37:23.160100 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:23.159580 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" containerID="cri-o://829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4" gracePeriod=30 Nov 01 09:37:28.430377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.430306 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:37:28.516675 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.516640 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.516904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.516745 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.516904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.516794 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.516904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.516856 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.516904 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.516894 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.517122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517026 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.517122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517090 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.517122 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517078 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:37:28.517282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517159 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:37:28.517282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517177 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.517282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517276 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket\") pod \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\" (UID: \"ea276ab2-0e20-4741-a42c-ccf8165de1cd\") " Nov 01 09:37:28.517484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517464 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:37:28.517569 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517548 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:37:28.517817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517790 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:37:28.517915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517818 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.517915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517839 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.517915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517854 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.517915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.517868 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.519333 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.519310 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:37:28.519459 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.519341 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "kube-api-access-475vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:37:28.519705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.519687 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:37:28.519817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.519709 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token" pod "ea276ab2-0e20-4741-a42c-ccf8165de1cd" (UID: "ea276ab2-0e20-4741-a42c-ccf8165de1cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:37:28.619985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.619934 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-475vp\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-kube-api-access-475vp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.619985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.619981 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.620278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.619998 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.620278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.620012 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.620278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.620027 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea276ab2-0e20-4741-a42c-ccf8165de1cd-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:37:28.684586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.684473 5710 generic.go:356] "Generic (PLEG): container finished" podID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerID="829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4" exitCode=0 Nov 01 09:37:28.684586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.684543 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" Nov 01 09:37:28.684586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.684545 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" event={"ID":"ea276ab2-0e20-4741-a42c-ccf8165de1cd","Type":"ContainerDied","Data":"829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4"} Nov 01 09:37:28.684914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.684605 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c" event={"ID":"ea276ab2-0e20-4741-a42c-ccf8165de1cd","Type":"ContainerDied","Data":"c58a13318938d7ecc452cfa704e34eb2ba1ababb7acfc6dccca23c842ad8926d"} Nov 01 09:37:28.684914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.684629 5710 scope.go:122] "RemoveContainer" containerID="829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4" Nov 01 09:37:28.696592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.696570 5710 scope.go:122] "RemoveContainer" containerID="3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba" Nov 01 09:37:28.706983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.706960 5710 scope.go:122] "RemoveContainer" containerID="829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4" Nov 01 09:37:28.707325 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:37:28.707299 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4\": container with ID starting with 829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4 not found: ID does not exist" containerID="829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4" Nov 01 09:37:28.707413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.707338 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4"} err="failed to get container status \"829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4\": rpc error: code = NotFound desc = could not find container \"829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4\": container with ID starting with 829c450b2a16241f23e385d0a72b5758592dce6702e9eef4af5cbdec321b74c4 not found: ID does not exist" Nov 01 09:37:28.707413 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.707364 5710 scope.go:122] "RemoveContainer" containerID="3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba" Nov 01 09:37:28.707619 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:37:28.707601 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba\": container with ID starting with 3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba not found: ID does not exist" containerID="3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba" Nov 01 09:37:28.707668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.707627 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba"} err="failed to get container status \"3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba\": rpc error: code = NotFound desc = could not find container \"3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba\": container with ID starting with 3d265dd82cb9d63b8b1b5ba32fd441993892dfd783a6033ad3121d2d5c07c4ba not found: ID does not exist" Nov 01 09:37:28.842758 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.842702 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c"] Nov 01 09:37:28.852155 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:28.852120 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e3728d-openshift-default-57f4445cb5-s696c"] Nov 01 09:37:29.060511 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:29.060426 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" path="/var/lib/kubelet/pods/ea276ab2-0e20-4741-a42c-ccf8165de1cd/volumes" Nov 01 09:37:30.014492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:37:30.014457 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:37:45.479557 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:37:45.479468 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:38:01.185772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:01.185710 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:38:01.202757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:01.202733 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:38:15.545694 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:38:15.545608 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:38:43.762578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.762544 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9"] Nov 01 09:38:43.763213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763190 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ee838c44-49b7-4a23-8268-bf09f6d00205" containerName="payload-pre-processing" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763217 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee838c44-49b7-4a23-8268-bf09f6d00205" containerName="payload-pre-processing" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763233 5710 cpu_manager.go:414] "containerMap: removing container" podUID="439e34dd-4059-4223-b2ba-52a4e44e0fd9" containerName="payload-processing" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763239 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439e34dd-4059-4223-b2ba-52a4e44e0fd9" containerName="payload-processing" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763251 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a63048c0-6615-43f9-909a-d79ee06fa42b" containerName="maas-api" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763255 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63048c0-6615-43f9-909a-d79ee06fa42b" containerName="maas-api" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763268 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" Nov 01 09:38:43.763269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763272 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763283 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763287 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763387 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763397 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee838c44-49b7-4a23-8268-bf09f6d00205" containerName="payload-pre-processing" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763404 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="439e34dd-4059-4223-b2ba-52a4e44e0fd9" containerName="payload-processing" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763409 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63048c0-6615-43f9-909a-d79ee06fa42b" containerName="maas-api" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763414 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a349c365-47f8-447e-aed4-288a8d4b67b9" containerName="istio-proxy" Nov 01 09:38:43.763509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.763421 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" Nov 01 09:38:43.767222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.767197 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.824640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.824592 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895pr\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.824640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.824641 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.824960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.824816 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.824960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.824845 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.824960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.824888 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.824960 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.824955 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.825176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.825010 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.825176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.825077 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.825176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.825121 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.868052 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.868015 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9"] Nov 01 09:38:43.926080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926040 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895pr\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926080 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926083 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926124 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926149 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926183 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926210 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926270 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926323 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926348 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.926952 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926929 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.927032 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.926967 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.927194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.927168 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.927325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.927214 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.927325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.927226 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.928833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.928811 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.929249 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.929225 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.984651 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.984607 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:43.984881 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:43.984734 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895pr\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rjfq9\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:44.081784 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:44.081661 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:44.513609 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:44.513467 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9"] Nov 01 09:38:44.517135 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:38:44.517099 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e69810_df19_492c_90bd_9d56ddc2d160.slice/crio-f65a754d2306930f39d4278c09493402c9240e34b5dbe11d6ef15bd486640d63 WatchSource:0}: Error finding container f65a754d2306930f39d4278c09493402c9240e34b5dbe11d6ef15bd486640d63: Status 404 returned error can't find the container with id f65a754d2306930f39d4278c09493402c9240e34b5dbe11d6ef15bd486640d63 Nov 01 09:38:44.519793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:44.519753 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:38:44.519913 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:44.519839 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:38:44.519913 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:44.519882 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:38:45.238277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:45.238237 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" event={"ID":"c5e69810-df19-492c-90bd-9d56ddc2d160","Type":"ContainerStarted","Data":"340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d"} Nov 01 09:38:45.238768 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:45.238290 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" event={"ID":"c5e69810-df19-492c-90bd-9d56ddc2d160","Type":"ContainerStarted","Data":"f65a754d2306930f39d4278c09493402c9240e34b5dbe11d6ef15bd486640d63"} Nov 01 09:38:45.369282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:45.369222 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podStartSLOduration=2.369207098 podStartE2EDuration="2.369207098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:38:45.366613192 +0000 UTC m=+3948.454949864" watchObservedRunningTime="2026-11-01 09:38:45.369207098 +0000 UTC m=+3948.457543769" Nov 01 09:38:45.478047 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:38:45.478008 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:38:46.087762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:46.084072 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:46.087762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:46.086232 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:46.087762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:46.086292 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:47.100744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:47.100232 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:47.100744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:47.100319 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:48.100748 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:48.096199 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:48.100748 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:48.096283 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:49.100746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:49.088166 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:49.100746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:49.088256 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:50.098743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:50.095163 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:50.098743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:50.095257 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:51.082967 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:51.082929 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:51.083407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:51.082984 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:52.082487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:52.082444 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 09:38:52.082693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:52.082517 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 09:38:54.096315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:54.096268 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:38:54.096783 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:54.096337 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:38:54.119594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:54.119553 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:54.120138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:54.120108 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:54.122045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:54.122027 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:38:58.503145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:58.503108 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:38:59.777236 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:59.777197 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w"] Nov 01 09:38:59.777664 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:38:59.777399 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" podUID="210ec782-1646-4570-a242-f458eec18a3b" containerName="maas-api" containerID="cri-o://a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4" gracePeriod=30 Nov 01 09:39:00.081546 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.081521 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:39:00.149874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.149832 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd\" (UniqueName: \"kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd\") pod \"210ec782-1646-4570-a242-f458eec18a3b\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " Nov 01 09:39:00.150038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.149946 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\") pod \"210ec782-1646-4570-a242-f458eec18a3b\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " Nov 01 09:39:00.150038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.150028 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca\") pod \"210ec782-1646-4570-a242-f458eec18a3b\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " Nov 01 09:39:00.150150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.150076 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs\") pod \"210ec782-1646-4570-a242-f458eec18a3b\" (UID: \"210ec782-1646-4570-a242-f458eec18a3b\") " Nov 01 09:39:00.150435 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.150406 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca" pod "210ec782-1646-4570-a242-f458eec18a3b" (UID: "210ec782-1646-4570-a242-f458eec18a3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:39:00.150892 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.150851 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ec782-1646-4570-a242-f458eec18a3b-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:00.167064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.166995 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls" pod "210ec782-1646-4570-a242-f458eec18a3b" (UID: "210ec782-1646-4570-a242-f458eec18a3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:39:00.169778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.169320 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd" pod "210ec782-1646-4570-a242-f458eec18a3b" (UID: "210ec782-1646-4570-a242-f458eec18a3b"). InnerVolumeSpecName "kube-api-access-cjhtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:39:00.170424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.170376 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs" pod "210ec782-1646-4570-a242-f458eec18a3b" (UID: "210ec782-1646-4570-a242-f458eec18a3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:39:00.252234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.252192 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhtd\" (UniqueName: \"kubernetes.io/projected/210ec782-1646-4570-a242-f458eec18a3b-kube-api-access-cjhtd\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:00.252234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.252230 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:00.252234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.252243 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210ec782-1646-4570-a242-f458eec18a3b-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:00.354228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.354189 5710 generic.go:356] "Generic (PLEG): container finished" podID="210ec782-1646-4570-a242-f458eec18a3b" containerID="a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4" exitCode=0 Nov 01 09:39:00.354455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.354250 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" Nov 01 09:39:00.354455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.354272 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" event={"ID":"210ec782-1646-4570-a242-f458eec18a3b","Type":"ContainerDied","Data":"a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4"} Nov 01 09:39:00.354455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.354361 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w" event={"ID":"210ec782-1646-4570-a242-f458eec18a3b","Type":"ContainerDied","Data":"1028bd3fee953b77d509591d066c9285fbafb479f4d1739937de78bf937b3502"} Nov 01 09:39:00.354455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.354409 5710 scope.go:122] "RemoveContainer" containerID="a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4" Nov 01 09:39:00.367150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.367126 5710 scope.go:122] "RemoveContainer" containerID="a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4" Nov 01 09:39:00.367493 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:39:00.367474 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4\": container with ID starting with a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4 not found: ID does not exist" containerID="a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4" Nov 01 09:39:00.367546 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.367503 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4"} err="failed to get container status \"a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4\": rpc error: code = NotFound desc = could not find container \"a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4\": container with ID starting with a7de6ad8c5ebf2ceb90a7dea70becab429454b017476c776d0654e63bfdd7cc4 not found: ID does not exist" Nov 01 09:39:00.509493 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.509451 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w"] Nov 01 09:39:00.516577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:00.516546 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8071b-6477d567bc-6jh8w"] Nov 01 09:39:01.059844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:01.059810 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210ec782-1646-4570-a242-f458eec18a3b" path="/var/lib/kubelet/pods/210ec782-1646-4570-a242-f458eec18a3b/volumes" Nov 01 09:39:11.239136 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.239088 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7"] Nov 01 09:39:11.239815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.239796 5710 cpu_manager.go:414] "containerMap: removing container" podUID="210ec782-1646-4570-a242-f458eec18a3b" containerName="maas-api" Nov 01 09:39:11.239895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.239816 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210ec782-1646-4570-a242-f458eec18a3b" containerName="maas-api" Nov 01 09:39:11.239895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.239843 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" Nov 01 09:39:11.239895 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.239850 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea276ab2-0e20-4741-a42c-ccf8165de1cd" containerName="istio-proxy" Nov 01 09:39:11.239995 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.239966 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="210ec782-1646-4570-a242-f458eec18a3b" containerName="maas-api" Nov 01 09:39:11.245035 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.245011 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.282822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.282787 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.283022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.283000 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrfp\" (UniqueName: \"kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp\") pod \"payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.343750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.343667 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-12a01a-dockercfg-2slmb\"" Nov 01 09:39:11.343750 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.343674 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-12a01a\"" Nov 01 09:39:11.359647 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.359597 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7"] Nov 01 09:39:11.386747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.384651 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrfp\" (UniqueName: \"kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp\") pod \"payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.386747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.384827 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.386747 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.385833 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.453938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.453888 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrfp\" (UniqueName: \"kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp\") pod \"payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.560879 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.560793 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:11.939578 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:11.939550 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7"] Nov 01 09:39:11.942511 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:39:11.942465 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc773e000_899a_44e0_bbb5_2014e54132df.slice/crio-9fddf243ee7b95c8e5b339639028fbb7d1805d166ebe688eac9308dc25e3a873 WatchSource:0}: Error finding container 9fddf243ee7b95c8e5b339639028fbb7d1805d166ebe688eac9308dc25e3a873: Status 404 returned error can't find the container with id 9fddf243ee7b95c8e5b339639028fbb7d1805d166ebe688eac9308dc25e3a873 Nov 01 09:39:12.476517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:12.476478 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" event={"ID":"c773e000-899a-44e0-bbb5-2014e54132df","Type":"ContainerStarted","Data":"396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20"} Nov 01 09:39:12.477157 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:12.476539 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" event={"ID":"c773e000-899a-44e0-bbb5-2014e54132df","Type":"ContainerStarted","Data":"9fddf243ee7b95c8e5b339639028fbb7d1805d166ebe688eac9308dc25e3a873"} Nov 01 09:39:12.477157 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:12.476569 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:12.588772 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:12.588694 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" podStartSLOduration=1.5886780520000001 podStartE2EDuration="1.588678052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:12.587711772 +0000 UTC m=+3975.676048444" watchObservedRunningTime="2026-11-01 09:39:12.588678052 +0000 UTC m=+3975.677014723" Nov 01 09:39:15.479246 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:39:15.479193 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:39:20.965880 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:20.965841 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n"] Nov 01 09:39:20.966933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:20.966103 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" containerID="cri-o://7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0" gracePeriod=30 Nov 01 09:39:22.295551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.295511 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf"] Nov 01 09:39:22.301660 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.301633 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.353175 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.353141 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49f18f-openshift-default-dockercfg-9smgr\"" Nov 01 09:39:22.478035 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.477261 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.478395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.478353 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.478643 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.478544 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.478896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.478875 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.479104 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.479058 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.479430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.479372 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.479823 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.479795 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.480104 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.480062 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p574f\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.480284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.480110 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.501818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.501768 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf"] Nov 01 09:39:22.581610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581515 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.581610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581565 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p574f\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.581610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581589 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581663 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581685 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581744 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581800 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581821 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.581863 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.582286 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582503 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.582338 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582731 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.582574 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582817 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.582800 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.582909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.582886 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.607762 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.606223 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.616788 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.616511 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.640604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.640569 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.640912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.640888 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p574f\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f\") pod \"e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:22.916734 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:22.916690 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:23.345533 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.345504 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf"] Nov 01 09:39:23.347577 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:39:23.347546 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182eb899_9d31_4514_b545_de771d99bc4b.slice/crio-38c65aeb0a83e27722a7b4f262d94bb9573b9b243b9afb07d1f66f9424e9a907 WatchSource:0}: Error finding container 38c65aeb0a83e27722a7b4f262d94bb9573b9b243b9afb07d1f66f9424e9a907: Status 404 returned error can't find the container with id 38c65aeb0a83e27722a7b4f262d94bb9573b9b243b9afb07d1f66f9424e9a907 Nov 01 09:39:23.350360 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.350325 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:39:23.350443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.350402 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:39:23.350443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.350431 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:39:23.487409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.487380 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:39:23.573249 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.573219 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" event={"ID":"182eb899-9d31-4514-b545-de771d99bc4b","Type":"ContainerStarted","Data":"725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14"} Nov 01 09:39:23.573464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.573267 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" event={"ID":"182eb899-9d31-4514-b545-de771d99bc4b","Type":"ContainerStarted","Data":"38c65aeb0a83e27722a7b4f262d94bb9573b9b243b9afb07d1f66f9424e9a907"} Nov 01 09:39:23.917842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:23.917806 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:24.924579 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:24.924545 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:25.085367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:25.085302 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podStartSLOduration=3.085282951 podStartE2EDuration="3.085282951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:23.736391889 +0000 UTC m=+3986.824728562" watchObservedRunningTime="2026-11-01 09:39:25.085282951 +0000 UTC m=+3988.173619624" Nov 01 09:39:25.588584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:25.588551 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:25.589443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:25.589426 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:39:26.241061 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.240993 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:39:26.335126 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335085 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335134 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335163 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335190 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335364 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335236 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335426 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335434 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:39:26.335597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335490 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335549 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335597 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335552 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:39:26.335874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335627 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x\") pod \"e1614416-7c0a-4515-bbcc-1be46726f747\" (UID: \"e1614416-7c0a-4515-bbcc-1be46726f747\") " Nov 01 09:39:26.335874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335823 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:39:26.335874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335860 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:39:26.336029 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.335946 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:39:26.336229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.336208 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.336366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.336234 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.336366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.336243 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.336366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.336252 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.336366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.336264 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1614416-7c0a-4515-bbcc-1be46726f747-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.338000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.337974 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "kube-api-access-xgk8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:39:26.338195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.338177 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:39:26.338312 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.338296 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:39:26.338373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.338323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo" pod "e1614416-7c0a-4515-bbcc-1be46726f747" (UID: "e1614416-7c0a-4515-bbcc-1be46726f747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:39:26.437950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.437908 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgk8x\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-kube-api-access-xgk8x\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.437950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.437942 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1614416-7c0a-4515-bbcc-1be46726f747-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.437950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.437954 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1614416-7c0a-4515-bbcc-1be46726f747-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.438215 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.437964 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1614416-7c0a-4515-bbcc-1be46726f747-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:39:26.597351 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.597312 5710 generic.go:356] "Generic (PLEG): container finished" podID="e1614416-7c0a-4515-bbcc-1be46726f747" containerID="7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0" exitCode=0 Nov 01 09:39:26.597557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.597375 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" Nov 01 09:39:26.597557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.597401 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" event={"ID":"e1614416-7c0a-4515-bbcc-1be46726f747","Type":"ContainerDied","Data":"7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0"} Nov 01 09:39:26.597557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.597470 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n" event={"ID":"e1614416-7c0a-4515-bbcc-1be46726f747","Type":"ContainerDied","Data":"86e69dfc3d5f602712c67b6bf15e3d491a4a17422f314317a6989b91886747bd"} Nov 01 09:39:26.597557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.597496 5710 scope.go:122] "RemoveContainer" containerID="7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0" Nov 01 09:39:26.610846 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.610824 5710 scope.go:122] "RemoveContainer" containerID="708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa" Nov 01 09:39:26.622219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.622194 5710 scope.go:122] "RemoveContainer" containerID="7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0" Nov 01 09:39:26.622554 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:39:26.622527 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0\": container with ID starting with 7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0 not found: ID does not exist" containerID="7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0" Nov 01 09:39:26.622655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.622564 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0"} err="failed to get container status \"7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0\": rpc error: code = NotFound desc = could not find container \"7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0\": container with ID starting with 7f3e7afb5941fc4bb1d301d5a921c3d24ee49ddc28b04659a9daa997090108c0 not found: ID does not exist" Nov 01 09:39:26.622655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.622587 5710 scope.go:122] "RemoveContainer" containerID="708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa" Nov 01 09:39:26.623029 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:39:26.623003 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa\": container with ID starting with 708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa not found: ID does not exist" containerID="708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa" Nov 01 09:39:26.623131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.623033 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa"} err="failed to get container status \"708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa\": rpc error: code = NotFound desc = could not find container \"708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa\": container with ID starting with 708eab7e5fdd3c07b53c1bed6c68c8dd0d16b49c531e5f9c4776ad204f7af2fa not found: ID does not exist" Nov 01 09:39:26.753117 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.753078 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n"] Nov 01 09:39:26.799535 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:26.799498 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b0d0f3-openshift-default-6dcd4597-5vf9n"] Nov 01 09:39:27.060592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:27.060502 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" path="/var/lib/kubelet/pods/e1614416-7c0a-4515-bbcc-1be46726f747/volumes" Nov 01 09:39:30.232830 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.232761 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv"] Nov 01 09:39:30.234868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.234841 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" Nov 01 09:39:30.235071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.235056 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" Nov 01 09:39:30.235316 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.235299 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" Nov 01 09:39:30.235426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.235415 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" Nov 01 09:39:30.235681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.235671 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" Nov 01 09:39:30.235793 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.235782 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1614416-7c0a-4515-bbcc-1be46726f747" containerName="istio-proxy" Nov 01 09:39:30.239758 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.239734 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv"] Nov 01 09:39:30.239896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.239851 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.289060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.289028 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49f18f-dockercfg-q4xj8\"" Nov 01 09:39:30.289237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.289206 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49f18f\"" Nov 01 09:39:30.305045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.304997 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22df\" (UniqueName: \"kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df\") pod \"payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.305274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.305252 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.406978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.406934 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.407214 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.407085 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22df\" (UniqueName: \"kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df\") pod \"payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.408738 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.408692 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.468387 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.468352 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22df\" (UniqueName: \"kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df\") pod \"payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.559456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.559357 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:30.939559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:30.939532 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv"] Nov 01 09:39:30.942485 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:39:30.942438 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e40014c_0133_482c_9ade_f87872de4999.slice/crio-e2c27986e472aaa2890397719a89ad56ccc1f0687efa6b7f63baa3393392af14 WatchSource:0}: Error finding container e2c27986e472aaa2890397719a89ad56ccc1f0687efa6b7f63baa3393392af14: Status 404 returned error can't find the container with id e2c27986e472aaa2890397719a89ad56ccc1f0687efa6b7f63baa3393392af14 Nov 01 09:39:31.638458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:31.638417 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" event={"ID":"2e40014c-0133-482c-9ade-f87872de4999","Type":"ContainerStarted","Data":"0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd"} Nov 01 09:39:31.638931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:31.638472 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" event={"ID":"2e40014c-0133-482c-9ade-f87872de4999","Type":"ContainerStarted","Data":"e2c27986e472aaa2890397719a89ad56ccc1f0687efa6b7f63baa3393392af14"} Nov 01 09:39:31.638931 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:31.638601 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:31.747836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:31.747769 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" podStartSLOduration=1.747752083 podStartE2EDuration="1.747752083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:39:31.746898551 +0000 UTC m=+3994.835235222" watchObservedRunningTime="2026-11-01 09:39:31.747752083 +0000 UTC m=+3994.836088754" Nov 01 09:39:42.647623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:42.647591 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:39:44.793786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:44.793752 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:39:45.548489 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:39:45.548454 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:39:59.348864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.348819 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd"] Nov 01 09:39:59.354736 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.354701 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.449140 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.449077 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd"] Nov 01 09:39:59.487317 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487277 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p748w\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487340 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487439 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487496 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487474 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487501 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487565 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487706 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487761 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.487836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.487795 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.588729 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.588665 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.588729 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.588730 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.588756 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.588787 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.588814 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.589282 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.589301 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589481 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.589432 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589481 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.589467 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.589595 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.589502 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.590239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.589973 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p748w\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.590239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.590055 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.590239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.590196 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.590499 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.590334 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.591755 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.591710 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.591901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.591883 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.648226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.648142 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.648226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.648209 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p748w\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vbndd\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:39:59.673024 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:39:59.672987 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:40:00.102216 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.102181 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd"] Nov 01 09:40:00.105040 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:40:00.104996 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89009dff_0b02_4d79_96a0_ac92c3bbe881.slice/crio-75500beb8454c1e176b02845d7e6716b15b4f433b7f211f6cbfd483c6977d4a3 WatchSource:0}: Error finding container 75500beb8454c1e176b02845d7e6716b15b4f433b7f211f6cbfd483c6977d4a3: Status 404 returned error can't find the container with id 75500beb8454c1e176b02845d7e6716b15b4f433b7f211f6cbfd483c6977d4a3 Nov 01 09:40:00.108043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.108010 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:40:00.108143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.108088 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:40:00.108143 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.108117 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:40:00.858686 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.858649 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" event={"ID":"89009dff-0b02-4d79-96a0-ac92c3bbe881","Type":"ContainerStarted","Data":"80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71"} Nov 01 09:40:00.859169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.858703 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" event={"ID":"89009dff-0b02-4d79-96a0-ac92c3bbe881","Type":"ContainerStarted","Data":"75500beb8454c1e176b02845d7e6716b15b4f433b7f211f6cbfd483c6977d4a3"} Nov 01 09:40:00.976966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:00.976906 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podStartSLOduration=1.97688924 podStartE2EDuration="1.97688924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:40:00.974879824 +0000 UTC m=+4024.063216497" watchObservedRunningTime="2026-11-01 09:40:00.97688924 +0000 UTC m=+4024.065225912" Nov 01 09:40:01.673272 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:01.673223 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:40:01.675074 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:01.675050 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:01.675159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:01.675104 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:02.673505 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:02.673468 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:02.674037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:02.673525 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:03.674185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:03.674141 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:03.674712 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:03.674228 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:04.673449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:04.673407 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:04.673697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:04.673484 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:05.673828 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:05.673782 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:05.674256 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:05.673854 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:06.674458 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:06.674415 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:06.674950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:06.674476 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:07.673689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:07.673648 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 01 09:40:07.673901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:07.673736 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 01 09:40:09.715927 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:09.715875 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:40:09.716469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:09.715961 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:40:09.828859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:09.828820 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:40:09.829367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:09.829322 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:40:09.831100 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:09.831080 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:40:15.479260 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:40:15.479222 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:40:45.479187 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:40:45.479145 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:40:46.564617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:40:46.564577 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:41:15.477832 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:41:15.477788 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:41:16.571565 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.571525 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2"] Nov 01 09:41:16.576159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.576133 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.625320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.625290 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c81a9d-openshift-default-dockercfg-wk2zk\"" Nov 01 09:41:16.675295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.675257 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2"] Nov 01 09:41:16.690786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.690748 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.690805 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.690905 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691022 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.691004 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.691108 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.691140 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691228 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.691182 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.691246 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.691402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.691283 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.808350 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.808434 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.808475 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.811382 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.816266 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.816325 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.816361 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.816457 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.816528 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.816556 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.818088 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.818356 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.819036 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.820407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.819594 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.825708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.825611 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.830993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.830959 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.887035 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.886999 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.891066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.891041 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n\") pod \"e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:16.892868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:16.892849 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:17.321823 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.321795 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2"] Nov 01 09:41:17.324475 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:41:17.324446 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53917f1_1232_480e_b014_e81651753c9a.slice/crio-137c87069dabac8cb32fcfd02addae0f33d96b314ddddffe1808a472521763a4 WatchSource:0}: Error finding container 137c87069dabac8cb32fcfd02addae0f33d96b314ddddffe1808a472521763a4: Status 404 returned error can't find the container with id 137c87069dabac8cb32fcfd02addae0f33d96b314ddddffe1808a472521763a4 Nov 01 09:41:17.327038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.327006 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:17.327111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.327075 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:17.327111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.327103 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:17.435586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.435540 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" event={"ID":"a53917f1-1232-480e-b014-e81651753c9a","Type":"ContainerStarted","Data":"19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3"} Nov 01 09:41:17.435822 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.435612 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" event={"ID":"a53917f1-1232-480e-b014-e81651753c9a","Type":"ContainerStarted","Data":"137c87069dabac8cb32fcfd02addae0f33d96b314ddddffe1808a472521763a4"} Nov 01 09:41:17.550486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.550416 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podStartSLOduration=1.550399207 podStartE2EDuration="1.550399207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:41:17.548937046 +0000 UTC m=+4100.637273730" watchObservedRunningTime="2026-11-01 09:41:17.550399207 +0000 UTC m=+4100.638735880" Nov 01 09:41:17.891054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.891010 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:41:17.893139 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:17.893111 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:18.899706 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:18.899673 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:19.453621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:19.452311 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:19.454125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:19.454085 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:41:26.630875 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:26.630838 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:41:26.631369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:26.630915 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:41:30.557390 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:30.557300 5710 generic.go:356] "Generic (PLEG): container finished" podID="182eb899-9d31-4514-b545-de771d99bc4b" containerID="725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14" exitCode=0 Nov 01 09:41:30.557390 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:30.557371 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" event={"ID":"182eb899-9d31-4514-b545-de771d99bc4b","Type":"ContainerDied","Data":"725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14"} Nov 01 09:41:30.557917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:30.557835 5710 scope.go:122] "RemoveContainer" containerID="725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14" Nov 01 09:41:30.559584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:30.559555 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:30.559771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:30.559639 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:30.559771 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:30.559689 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:31.567774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:31.567730 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" event={"ID":"182eb899-9d31-4514-b545-de771d99bc4b","Type":"ContainerStarted","Data":"bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df"} Nov 01 09:41:31.917095 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:31.917050 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:41:31.917350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:31.917325 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 09:41:31.917441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:31.917381 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 09:41:32.917689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:32.917183 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 09:41:32.917689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:32.917252 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 09:41:33.084877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.081147 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn"] Nov 01 09:41:33.112062 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.112029 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.160902 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.160869 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b3d6bb-openshift-default-dockercfg-rvv5c\"" Nov 01 09:41:33.217534 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217432 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217534 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217510 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217546 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217580 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46d6\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217630 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217656 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217708 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217845 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.217897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.217878 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.221514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.221484 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn"] Nov 01 09:41:33.318951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.318914 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319054 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319121 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319150 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319178 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46d6\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319242 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319270 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319320 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319638 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319418 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319655 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319702 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319686 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.319845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.319797 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.320067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.320029 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.320473 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.320446 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.321839 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.321821 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.322151 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.322131 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.408744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.408424 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46d6\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.409363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.409335 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token\") pod \"e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.444776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.441746 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:33.917939 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.917902 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 09:41:33.918378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.917954 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 09:41:33.937069 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.937035 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn"] Nov 01 09:41:33.940600 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:41:33.940570 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8622048f_3e2c_462d_bbd7_b1359d75a387.slice/crio-21bc061e666276b0ea19c9e5a060c499e541c618be4e3eb472e94880c03678dc WatchSource:0}: Error finding container 21bc061e666276b0ea19c9e5a060c499e541c618be4e3eb472e94880c03678dc: Status 404 returned error can't find the container with id 21bc061e666276b0ea19c9e5a060c499e541c618be4e3eb472e94880c03678dc Nov 01 09:41:33.943346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.943307 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:33.943457 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.943380 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:33.943457 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:33.943411 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:41:34.628663 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:34.628623 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" event={"ID":"8622048f-3e2c-462d-bbd7-b1359d75a387","Type":"ContainerStarted","Data":"8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a"} Nov 01 09:41:34.628877 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:34.628698 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" event={"ID":"8622048f-3e2c-462d-bbd7-b1359d75a387","Type":"ContainerStarted","Data":"21bc061e666276b0ea19c9e5a060c499e541c618be4e3eb472e94880c03678dc"} Nov 01 09:41:34.811527 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:34.811453 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podStartSLOduration=1.811429712 podStartE2EDuration="1.811429712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:41:34.807256021 +0000 UTC m=+4117.895592697" watchObservedRunningTime="2026-11-01 09:41:34.811429712 +0000 UTC m=+4117.899766386" Nov 01 09:41:34.922008 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:34.921700 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 09:41:34.922008 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:34.921793 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 09:41:35.445906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:35.445872 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:35.452039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:35.452009 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:35.635001 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:35.634973 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:35.635907 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:35.635881 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:41:35.918063 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:35.918024 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 09:41:35.918289 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:35.918084 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 09:41:36.917353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:36.917316 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 09:41:36.917868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:36.917373 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 09:41:37.922339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:37.922306 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:41:37.922776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:37.922759 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:41:37.923761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:37.923709 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:41:41.061039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.061001 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n"] Nov 01 09:41:41.065704 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.065678 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.117196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.117158 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n"] Nov 01 09:41:41.160701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.160653 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77c2\" (UniqueName: \"kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2\") pod \"payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.161146 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.161118 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.162894 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.162871 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b3d6bb\"" Nov 01 09:41:41.163361 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.163273 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b3d6bb-dockercfg-xjvdt\"" Nov 01 09:41:41.262826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.262785 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.263395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.263329 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77c2\" (UniqueName: \"kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2\") pod \"payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.265389 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.265316 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.322454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.322372 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77c2\" (UniqueName: \"kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2\") pod \"payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.381352 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.381311 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:41.766047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:41.766016 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n"] Nov 01 09:41:41.768619 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:41:41.768586 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1277329_4de9_4ba2_bea4_f0c9d2cc5cdd.slice/crio-304a24d8f09f95ea42b4a2b7f4dda9f7cc240933afcfe65cb8390414c21cd063 WatchSource:0}: Error finding container 304a24d8f09f95ea42b4a2b7f4dda9f7cc240933afcfe65cb8390414c21cd063: Status 404 returned error can't find the container with id 304a24d8f09f95ea42b4a2b7f4dda9f7cc240933afcfe65cb8390414c21cd063 Nov 01 09:41:42.684602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:42.684564 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" event={"ID":"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd","Type":"ContainerStarted","Data":"6194ea206f2c41c02055cbd5c736003772eb0ef733ff51842dc3eae55140b96e"} Nov 01 09:41:42.685130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:42.684620 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" event={"ID":"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd","Type":"ContainerStarted","Data":"304a24d8f09f95ea42b4a2b7f4dda9f7cc240933afcfe65cb8390414c21cd063"} Nov 01 09:41:42.685130 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:42.684704 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:41:42.796107 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:42.796052 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" podStartSLOduration=1.796034737 podStartE2EDuration="1.796034737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:41:42.79252789 +0000 UTC m=+4125.880864562" watchObservedRunningTime="2026-11-01 09:41:42.796034737 +0000 UTC m=+4125.884371434" Nov 01 09:41:45.477932 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:41:45.477892 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:41:53.695639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:41:53.695607 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:42:15.466941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.466904 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z"] Nov 01 09:42:15.471845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.471820 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.476215 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.476186 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.476380 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.476270 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hst2\" (UniqueName: \"kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2\") pod \"payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.484864 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:42:15.484835 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:42:15.543362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.543310 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z"] Nov 01 09:42:15.580402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.580367 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f41050-dockercfg-xhx8m\"" Nov 01 09:42:15.589086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.589039 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f41050\"" Nov 01 09:42:15.701981 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.637832 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hst2\" (UniqueName: \"kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2\") pod \"payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.713238 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.713014 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.714706 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.714672 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7"] Nov 01 09:42:15.720417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.720346 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.735103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.735023 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hst2\" (UniqueName: \"kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2\") pod \"payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.735974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.735325 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:15.736270 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.735198 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7"] Nov 01 09:42:15.788380 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.788339 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:15.815278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.815247 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:15.815451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.815314 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwv9w\" (UniqueName: \"kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w\") pod \"payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:15.918184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.918127 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:15.918421 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.918249 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwv9w\" (UniqueName: \"kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w\") pod \"payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:15.926662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.919558 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:15.977039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:15.976960 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwv9w\" (UniqueName: \"kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w\") pod \"payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:16.054864 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.054833 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:16.167761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.167708 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z"] Nov 01 09:42:16.169214 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:42:16.169179 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a3c8fe_6345_4675_8883_46208026215c.slice/crio-063af15987fcc428fc9e150d7f00514e0696ecdd213005413c6c1f0dce949519 WatchSource:0}: Error finding container 063af15987fcc428fc9e150d7f00514e0696ecdd213005413c6c1f0dce949519: Status 404 returned error can't find the container with id 063af15987fcc428fc9e150d7f00514e0696ecdd213005413c6c1f0dce949519 Nov 01 09:42:16.436454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.436427 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7"] Nov 01 09:42:16.439459 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:42:16.439422 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56390f5e_1626_437c_bbc8_4248040edacb.slice/crio-8c5e70b62e08aef37d4d5a8873dcee60a64d8d0fc7dfe458e20d6ab061383f57 WatchSource:0}: Error finding container 8c5e70b62e08aef37d4d5a8873dcee60a64d8d0fc7dfe458e20d6ab061383f57: Status 404 returned error can't find the container with id 8c5e70b62e08aef37d4d5a8873dcee60a64d8d0fc7dfe458e20d6ab061383f57 Nov 01 09:42:16.925954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.925915 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" event={"ID":"54a3c8fe-6345-4675-8883-46208026215c","Type":"ContainerStarted","Data":"38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8"} Nov 01 09:42:16.926426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.925975 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" event={"ID":"54a3c8fe-6345-4675-8883-46208026215c","Type":"ContainerStarted","Data":"063af15987fcc428fc9e150d7f00514e0696ecdd213005413c6c1f0dce949519"} Nov 01 09:42:16.926426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.926002 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:16.927774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.927751 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" event={"ID":"56390f5e-1626-437c-bbc8-4248040edacb","Type":"ContainerStarted","Data":"2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1"} Nov 01 09:42:16.927916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.927789 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" event={"ID":"56390f5e-1626-437c-bbc8-4248040edacb","Type":"ContainerStarted","Data":"8c5e70b62e08aef37d4d5a8873dcee60a64d8d0fc7dfe458e20d6ab061383f57"} Nov 01 09:42:16.927916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:16.927856 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:17.036751 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:17.036661 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" podStartSLOduration=2.036647537 podStartE2EDuration="2.036647537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:42:17.034448467 +0000 UTC m=+4160.122785142" watchObservedRunningTime="2026-11-01 09:42:17.036647537 +0000 UTC m=+4160.124984203" Nov 01 09:42:17.143985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:17.143916 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" podStartSLOduration=2.143900376 podStartE2EDuration="2.143900376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:42:17.14330835 +0000 UTC m=+4160.231645021" watchObservedRunningTime="2026-11-01 09:42:17.143900376 +0000 UTC m=+4160.232237110" Nov 01 09:42:27.938225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:27.938191 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:42:27.938837 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:27.938274 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:42:36.751382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:36.751344 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv"] Nov 01 09:42:41.752492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:41.752419 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" podUID="2e40014c-0133-482c-9ade-f87872de4999" containerName="payload-processing" containerID="cri-o://0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd" gracePeriod=25 Nov 01 09:42:42.030332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.030304 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:42:42.128102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.128053 5710 generic.go:356] "Generic (PLEG): container finished" podID="2e40014c-0133-482c-9ade-f87872de4999" containerID="0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd" exitCode=0 Nov 01 09:42:42.128301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.128106 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" Nov 01 09:42:42.128301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.128144 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" event={"ID":"2e40014c-0133-482c-9ade-f87872de4999","Type":"ContainerDied","Data":"0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd"} Nov 01 09:42:42.128301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.128207 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv" event={"ID":"2e40014c-0133-482c-9ade-f87872de4999","Type":"ContainerDied","Data":"e2c27986e472aaa2890397719a89ad56ccc1f0687efa6b7f63baa3393392af14"} Nov 01 09:42:42.128301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.128240 5710 scope.go:122] "RemoveContainer" containerID="0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd" Nov 01 09:42:42.139310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.138903 5710 scope.go:122] "RemoveContainer" containerID="0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd" Nov 01 09:42:42.139432 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:42:42.139344 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd\": container with ID starting with 0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd not found: ID does not exist" containerID="0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd" Nov 01 09:42:42.139432 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.139381 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd"} err="failed to get container status \"0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd\": rpc error: code = NotFound desc = could not find container \"0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd\": container with ID starting with 0eb7cc99fbe01dc74ac7c6af0832d10c43c5f9728a533fad0508c5fc247881cd not found: ID does not exist" Nov 01 09:42:42.205524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.205482 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df\" (UniqueName: \"kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df\") pod \"2e40014c-0133-482c-9ade-f87872de4999\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " Nov 01 09:42:42.205757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.205615 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume\") pod \"2e40014c-0133-482c-9ade-f87872de4999\" (UID: \"2e40014c-0133-482c-9ade-f87872de4999\") " Nov 01 09:42:42.206079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.206048 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume" pod "2e40014c-0133-482c-9ade-f87872de4999" (UID: "2e40014c-0133-482c-9ade-f87872de4999"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:42:42.206483 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.206461 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e40014c-0133-482c-9ade-f87872de4999-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:42:42.208221 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.208198 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df" pod "2e40014c-0133-482c-9ade-f87872de4999" (UID: "2e40014c-0133-482c-9ade-f87872de4999"). InnerVolumeSpecName "kube-api-access-b22df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:42:42.308322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.308242 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b22df\" (UniqueName: \"kubernetes.io/projected/2e40014c-0133-482c-9ade-f87872de4999-kube-api-access-b22df\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:42:42.543766 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.543707 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv"] Nov 01 09:42:42.547847 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:42.547815 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49f18f-c5bbc7f57-xnnsv"] Nov 01 09:42:43.059773 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:42:43.059741 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e40014c-0133-482c-9ade-f87872de4999" path="/var/lib/kubelet/pods/2e40014c-0133-482c-9ade-f87872de4999/volumes" Nov 01 09:42:45.478599 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:42:45.478556 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:43:01.234325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:01.234290 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:43:01.250524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:01.250497 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:43:12.315804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:12.315765 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:43:15.479360 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:43:15.479320 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:43:20.494911 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:20.494864 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:43:20.495384 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:20.494952 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:43:23.489778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:23.489743 5710 generic.go:356] "Generic (PLEG): container finished" podID="a53917f1-1232-480e-b014-e81651753c9a" containerID="19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3" exitCode=0 Nov 01 09:43:23.490294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:23.489748 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" event={"ID":"a53917f1-1232-480e-b014-e81651753c9a","Type":"ContainerDied","Data":"19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3"} Nov 01 09:43:23.490294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:23.490277 5710 scope.go:122] "RemoveContainer" containerID="19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3" Nov 01 09:43:23.492158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:23.492129 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:43:23.492260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:23.492203 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:43:23.492260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:23.492238 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:43:24.514172 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:24.514132 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" event={"ID":"a53917f1-1232-480e-b014-e81651753c9a","Type":"ContainerStarted","Data":"4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214"} Nov 01 09:43:24.903743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:24.894140 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:43:24.903743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:24.894270 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 09:43:24.903743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:24.894318 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 09:43:25.902749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:25.901775 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 09:43:25.902749 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:25.901847 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 09:43:26.898887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:26.898845 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 09:43:26.899233 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:26.899186 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 09:43:27.901745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:27.901184 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 09:43:27.901745 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:27.901256 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 09:43:28.903746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:28.897709 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 09:43:28.903746 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:28.902848 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 09:43:29.894073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:29.894039 5710 patch_prober.go:29] interesting pod/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 09:43:29.894274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:29.894097 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 09:43:30.899174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:30.899141 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:43:30.899633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:30.899437 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:43:30.900195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:30.900179 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:43:45.589342 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:43:45.589231 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:43:45.707985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:45.707946 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:43:49.931884 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:49.931847 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh"] Nov 01 09:43:49.932610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:49.932595 5710 cpu_manager.go:414] "containerMap: removing container" podUID="2e40014c-0133-482c-9ade-f87872de4999" containerName="payload-processing" Nov 01 09:43:49.932669 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:49.932612 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e40014c-0133-482c-9ade-f87872de4999" containerName="payload-processing" Nov 01 09:43:49.932776 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:49.932766 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e40014c-0133-482c-9ade-f87872de4999" containerName="payload-processing" Nov 01 09:43:49.936914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:49.936890 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.034668 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.034635 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f41050-serving-cert\"" Nov 01 09:43:50.037782 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.037756 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh"] Nov 01 09:43:50.045385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.045362 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.045489 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.045401 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.045563 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.045551 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.045610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.045580 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cz72\" (UniqueName: \"kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.146938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.146893 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.147148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.146946 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cz72\" (UniqueName: \"kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.147148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.147040 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.147148 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.147065 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.147295 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:43:50.147213 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-serving-cert: secret "maas-api-e2e-inf-b-f41050-serving-cert" not found Nov 01 09:43:50.147295 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:43:50.147277 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls podName:4336fec2-f181-4506-bca4-283fa60af703 nodeName:}" failed. No retries permitted until 2026-11-01 09:43:50.64726238 +0000 UTC m=+4253.735599043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls") pod "maas-api-e2e-inf-b-f41050-8584d64769-569bh" (UID: "4336fec2-f181-4506-bca4-283fa60af703") : secret "maas-api-e2e-inf-b-f41050-serving-cert" not found Nov 01 09:43:50.147787 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.147765 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.149802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.149779 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.236679 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.236566 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cz72\" (UniqueName: \"kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.661463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.661421 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.664261 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.664229 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\") pod \"maas-api-e2e-inf-b-f41050-8584d64769-569bh\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:50.852354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:50.852314 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:51.254354 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:51.254324 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh"] Nov 01 09:43:51.256603 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:43:51.256571 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4336fec2_f181_4506_bca4_283fa60af703.slice/crio-c0856905aab5f7a0394172886c4fdc12ba479b01b29e9d02a93ba299e82d6333 WatchSource:0}: Error finding container c0856905aab5f7a0394172886c4fdc12ba479b01b29e9d02a93ba299e82d6333: Status 404 returned error can't find the container with id c0856905aab5f7a0394172886c4fdc12ba479b01b29e9d02a93ba299e82d6333 Nov 01 09:43:51.258055 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:51.258035 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:43:51.749656 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:51.749617 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" event={"ID":"4336fec2-f181-4506-bca4-283fa60af703","Type":"ContainerStarted","Data":"c0856905aab5f7a0394172886c4fdc12ba479b01b29e9d02a93ba299e82d6333"} Nov 01 09:43:52.758490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:52.758450 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" event={"ID":"4336fec2-f181-4506-bca4-283fa60af703","Type":"ContainerStarted","Data":"33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d"} Nov 01 09:43:52.759037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:52.758566 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:52.872169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:52.872090 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" podStartSLOduration=2.902512233 podStartE2EDuration="3.872067713s" totalImagesPullingTime="969.55548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:43:52.868288127 +0000 UTC m=+4255.956624799" watchObservedRunningTime="2026-11-01 09:43:52.872067713 +0000 UTC m=+4255.960404380" Nov 01 09:43:55.782405 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:55.782301 5710 generic.go:356] "Generic (PLEG): container finished" podID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerID="8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a" exitCode=0 Nov 01 09:43:55.782405 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:55.782373 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" event={"ID":"8622048f-3e2c-462d-bbd7-b1359d75a387","Type":"ContainerDied","Data":"8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a"} Nov 01 09:43:55.782961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:55.782853 5710 scope.go:122] "RemoveContainer" containerID="8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a" Nov 01 09:43:55.784611 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:55.784583 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:43:55.784695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:55.784657 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:43:55.784695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:55.784692 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:43:56.814108 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:56.814063 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" event={"ID":"8622048f-3e2c-462d-bbd7-b1359d75a387","Type":"ContainerStarted","Data":"0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a"} Nov 01 09:43:57.445792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:57.445756 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:43:57.446033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:57.446014 5710 patch_prober.go:29] interesting pod/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 09:43:57.446084 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:57.446070 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 09:43:58.445653 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:58.445613 5710 patch_prober.go:29] interesting pod/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 09:43:58.446154 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:58.445670 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 09:43:58.774060 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:58.773965 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:43:59.445455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:59.445423 5710 patch_prober.go:29] interesting pod/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 09:43:59.445661 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:43:59.445492 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 09:44:00.446194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:00.446162 5710 patch_prober.go:29] interesting pod/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 09:44:00.446617 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:00.446228 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 09:44:01.445778 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:01.445734 5710 patch_prober.go:29] interesting pod/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 01 09:44:01.446013 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:01.445806 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 01 09:44:02.451363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:02.451330 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:44:02.451849 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:02.451814 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:44:02.452789 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:02.452765 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:44:15.481007 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:44:15.480959 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:44:41.636120 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:44:41.636083 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:44:45.478969 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:44:45.478916 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:45:00.222686 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.222637 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr"] Nov 01 09:45:00.228295 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.228269 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt"] Nov 01 09:45:00.228451 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.228384 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.232582 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.232559 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.251606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.251575 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.251833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.251641 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.251833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.251803 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstnk\" (UniqueName: \"kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.251985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.251876 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlld4\" (UniqueName: \"kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.252325 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.252305 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw"] Nov 01 09:45:00.258739 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.258698 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.267797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.267768 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8"] Nov 01 09:45:00.273079 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.273054 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.281217 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.281173 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qwxrs\"" Nov 01 09:45:00.325045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.325010 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr"] Nov 01 09:45:00.328857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.328823 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt"] Nov 01 09:45:00.330619 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.330593 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw"] Nov 01 09:45:00.354480 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.354442 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstnk\" (UniqueName: \"kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.354814 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.354787 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlld4\" (UniqueName: \"kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.355056 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.355033 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hp5\" (UniqueName: \"kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.355369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.355241 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.355548 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.355505 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.355684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.355560 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nlj\" (UniqueName: \"kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj\") pod \"maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.355684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.355636 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.355850 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.355698 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.356346 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.356314 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.356680 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.356656 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.360184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.360161 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8"] Nov 01 09:45:00.413147 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.413110 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstnk\" (UniqueName: \"kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk\") pod \"maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.413341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.413158 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlld4\" (UniqueName: \"kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4\") pod \"maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.457075 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.457035 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nlj\" (UniqueName: \"kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj\") pod \"maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.457303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.457099 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.457303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.457207 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hp5\" (UniqueName: \"kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.457303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.457247 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.457933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.457905 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.458045 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.457937 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.515110 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.515032 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nlj\" (UniqueName: \"kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj\") pod \"maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.515272 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.515120 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hp5\" (UniqueName: \"kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.545929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.545895 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:00.555588 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.555558 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:00.576602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.576569 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:00.588484 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.588333 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:00.939222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.939187 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr"] Nov 01 09:45:00.941680 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:45:00.941644 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d1dfbc_e215_43af_9b9b_e9602ab9ea0e.slice/crio-a1776b0bbe5cf39310b0a2737e3adb5803dae40cb46aeb31b6ac487dbe98d113 WatchSource:0}: Error finding container a1776b0bbe5cf39310b0a2737e3adb5803dae40cb46aeb31b6ac487dbe98d113: Status 404 returned error can't find the container with id a1776b0bbe5cf39310b0a2737e3adb5803dae40cb46aeb31b6ac487dbe98d113 Nov 01 09:45:00.981033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:00.981006 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt"] Nov 01 09:45:00.983289 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:45:00.983259 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ca7164_be67_45b6_b5f2_b89535f0f70e.slice/crio-12393f8e59be345bc29817c3ddabb17114e041737a19442efe0a5834187121a2 WatchSource:0}: Error finding container 12393f8e59be345bc29817c3ddabb17114e041737a19442efe0a5834187121a2: Status 404 returned error can't find the container with id 12393f8e59be345bc29817c3ddabb17114e041737a19442efe0a5834187121a2 Nov 01 09:45:01.002020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.001995 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw"] Nov 01 09:45:01.004428 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:45:01.004402 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b987afa_970c_4e07_93f2_a567ad2a7f2a.slice/crio-b4272b73cb305e251e48fb6906e70c9a0f3c5b001dc974614b73b426c6394785 WatchSource:0}: Error finding container b4272b73cb305e251e48fb6906e70c9a0f3c5b001dc974614b73b426c6394785: Status 404 returned error can't find the container with id b4272b73cb305e251e48fb6906e70c9a0f3c5b001dc974614b73b426c6394785 Nov 01 09:45:01.031033 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.031005 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8"] Nov 01 09:45:01.033248 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:45:01.033221 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34951a7b_c05a_4e28_ad67_318a376b1616.slice/crio-ed116dbf00d83aa3de8efa9afa750b6ca0707b30d84c7669ca2ab9707faed4b8 WatchSource:0}: Error finding container ed116dbf00d83aa3de8efa9afa750b6ca0707b30d84c7669ca2ab9707faed4b8: Status 404 returned error can't find the container with id ed116dbf00d83aa3de8efa9afa750b6ca0707b30d84c7669ca2ab9707faed4b8 Nov 01 09:45:01.316480 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.316427 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerStarted","Data":"39d2a3f5a2a8d04ec1c891a3f2fa213783b319e9af202137757e192488b21537"} Nov 01 09:45:01.317030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.316499 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerStarted","Data":"ed116dbf00d83aa3de8efa9afa750b6ca0707b30d84c7669ca2ab9707faed4b8"} Nov 01 09:45:01.317908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.317881 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerStarted","Data":"a1776b0bbe5cf39310b0a2737e3adb5803dae40cb46aeb31b6ac487dbe98d113"} Nov 01 09:45:01.319229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.319191 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerStarted","Data":"12393f8e59be345bc29817c3ddabb17114e041737a19442efe0a5834187121a2"} Nov 01 09:45:01.320592 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.320571 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerStarted","Data":"b4272b73cb305e251e48fb6906e70c9a0f3c5b001dc974614b73b426c6394785"} Nov 01 09:45:01.426677 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:01.426624 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" podStartSLOduration=1.4266034969999999 podStartE2EDuration="1.426603497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:01.42646489 +0000 UTC m=+4324.514801561" watchObservedRunningTime="2026-11-01 09:45:01.426603497 +0000 UTC m=+4324.514940162" Nov 01 09:45:02.330171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.330135 5710 generic.go:356] "Generic (PLEG): container finished" podID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerID="c60291c423090da18a1f750d0004cf558a3e467fdc7028c6f65f4043ab9dabd5" exitCode=60 Nov 01 09:45:02.330665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.330227 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerDied","Data":"c60291c423090da18a1f750d0004cf558a3e467fdc7028c6f65f4043ab9dabd5"} Nov 01 09:45:02.330665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.330584 5710 scope.go:122] "RemoveContainer" containerID="c60291c423090da18a1f750d0004cf558a3e467fdc7028c6f65f4043ab9dabd5" Nov 01 09:45:02.332202 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.332178 5710 generic.go:356] "Generic (PLEG): container finished" podID="34951a7b-c05a-4e28-ad67-318a376b1616" containerID="39d2a3f5a2a8d04ec1c891a3f2fa213783b319e9af202137757e192488b21537" exitCode=60 Nov 01 09:45:02.332345 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.332298 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerDied","Data":"39d2a3f5a2a8d04ec1c891a3f2fa213783b319e9af202137757e192488b21537"} Nov 01 09:45:02.332661 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.332638 5710 scope.go:122] "RemoveContainer" containerID="39d2a3f5a2a8d04ec1c891a3f2fa213783b319e9af202137757e192488b21537" Nov 01 09:45:02.334292 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.334268 5710 generic.go:356] "Generic (PLEG): container finished" podID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerID="5a481a3cc6680dfd7244bde756a067eb084be4838dadb39b17858c1ab534652d" exitCode=60 Nov 01 09:45:02.334420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.334346 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerDied","Data":"5a481a3cc6680dfd7244bde756a067eb084be4838dadb39b17858c1ab534652d"} Nov 01 09:45:02.334685 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.334642 5710 scope.go:122] "RemoveContainer" containerID="5a481a3cc6680dfd7244bde756a067eb084be4838dadb39b17858c1ab534652d" Nov 01 09:45:02.336301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.336229 5710 generic.go:356] "Generic (PLEG): container finished" podID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerID="8e3802d42adea48edcd1530b2df2c93c3ea72a6d4069fc5e81f61014c11f9710" exitCode=60 Nov 01 09:45:02.336301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.336276 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerDied","Data":"8e3802d42adea48edcd1530b2df2c93c3ea72a6d4069fc5e81f61014c11f9710"} Nov 01 09:45:02.336577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:02.336562 5710 scope.go:122] "RemoveContainer" containerID="8e3802d42adea48edcd1530b2df2c93c3ea72a6d4069fc5e81f61014c11f9710" Nov 01 09:45:03.346485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.346446 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerStarted","Data":"681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d"} Nov 01 09:45:03.348950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.348913 5710 generic.go:356] "Generic (PLEG): container finished" podID="34951a7b-c05a-4e28-ad67-318a376b1616" containerID="78c914bff14fb29af6644f40c9055b01918d919ca920544454677d871bd547de" exitCode=60 Nov 01 09:45:03.349162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.349023 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerDied","Data":"78c914bff14fb29af6644f40c9055b01918d919ca920544454677d871bd547de"} Nov 01 09:45:03.349162 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.349079 5710 scope.go:122] "RemoveContainer" containerID="39d2a3f5a2a8d04ec1c891a3f2fa213783b319e9af202137757e192488b21537" Nov 01 09:45:03.349577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.349556 5710 scope.go:122] "RemoveContainer" containerID="78c914bff14fb29af6644f40c9055b01918d919ca920544454677d871bd547de" Nov 01 09:45:03.350026 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:03.349995 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8_odh-ai-gateway-infra(34951a7b-c05a-4e28-ad67-318a376b1616)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" Nov 01 09:45:03.352161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.352133 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerStarted","Data":"0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10"} Nov 01 09:45:03.354076 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.354052 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerStarted","Data":"b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4"} Nov 01 09:45:03.459016 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.458959 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" podStartSLOduration=3.458943635 podStartE2EDuration="3.458943635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.457530228 +0000 UTC m=+4326.545866912" watchObservedRunningTime="2026-11-01 09:45:03.458943635 +0000 UTC m=+4326.547280310" Nov 01 09:45:03.674504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.674442 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" podStartSLOduration=3.67442678 podStartE2EDuration="3.67442678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.673217701 +0000 UTC m=+4326.761554373" watchObservedRunningTime="2026-11-01 09:45:03.67442678 +0000 UTC m=+4326.762763451" Nov 01 09:45:03.783486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:03.783423 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" podStartSLOduration=3.7834071910000002 podStartE2EDuration="3.783407191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:45:03.780953203 +0000 UTC m=+4326.869289874" watchObservedRunningTime="2026-11-01 09:45:03.783407191 +0000 UTC m=+4326.871743924" Nov 01 09:45:04.362959 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.362869 5710 generic.go:356] "Generic (PLEG): container finished" podID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerID="681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d" exitCode=60 Nov 01 09:45:04.363482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.362961 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerDied","Data":"681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d"} Nov 01 09:45:04.363482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.363030 5710 scope.go:122] "RemoveContainer" containerID="c60291c423090da18a1f750d0004cf558a3e467fdc7028c6f65f4043ab9dabd5" Nov 01 09:45:04.363482 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.363286 5710 scope.go:122] "RemoveContainer" containerID="681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d" Nov 01 09:45:04.363687 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:04.363661 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw_odh-ai-gateway-infra(3b987afa-970c-4e07-93f2-a567ad2a7f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" Nov 01 09:45:04.365359 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.365332 5710 scope.go:122] "RemoveContainer" containerID="78c914bff14fb29af6644f40c9055b01918d919ca920544454677d871bd547de" Nov 01 09:45:04.365787 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:04.365702 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8_odh-ai-gateway-infra(34951a7b-c05a-4e28-ad67-318a376b1616)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" Nov 01 09:45:04.367110 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.367088 5710 generic.go:356] "Generic (PLEG): container finished" podID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerID="0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10" exitCode=60 Nov 01 09:45:04.367229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.367149 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerDied","Data":"0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10"} Nov 01 09:45:04.367477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.367463 5710 scope.go:122] "RemoveContainer" containerID="0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10" Nov 01 09:45:04.367860 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:04.367834 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr_odh-ai-gateway-infra(58d1dfbc-e215-43af-9b9b-e9602ab9ea0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" Nov 01 09:45:04.369487 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.369461 5710 generic.go:356] "Generic (PLEG): container finished" podID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerID="b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4" exitCode=60 Nov 01 09:45:04.369586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.369515 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerDied","Data":"b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4"} Nov 01 09:45:04.369855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.369840 5710 scope.go:122] "RemoveContainer" containerID="b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4" Nov 01 09:45:04.370155 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:04.370137 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt_odh-ai-gateway-infra(47ca7164-be67-45b6-b5f2-b89535f0f70e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" Nov 01 09:45:04.377925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.377899 5710 scope.go:122] "RemoveContainer" containerID="5a481a3cc6680dfd7244bde756a067eb084be4838dadb39b17858c1ab534652d" Nov 01 09:45:04.393774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:04.393746 5710 scope.go:122] "RemoveContainer" containerID="8e3802d42adea48edcd1530b2df2c93c3ea72a6d4069fc5e81f61014c11f9710" Nov 01 09:45:05.378983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:05.378956 5710 scope.go:122] "RemoveContainer" containerID="b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4" Nov 01 09:45:05.379537 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:05.379330 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt_odh-ai-gateway-infra(47ca7164-be67-45b6-b5f2-b89535f0f70e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" Nov 01 09:45:05.380953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:05.380932 5710 scope.go:122] "RemoveContainer" containerID="681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d" Nov 01 09:45:05.381285 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:05.381259 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw_odh-ai-gateway-infra(3b987afa-970c-4e07-93f2-a567ad2a7f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" Nov 01 09:45:05.382581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:05.382565 5710 scope.go:122] "RemoveContainer" containerID="0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10" Nov 01 09:45:05.382870 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:05.382850 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr_odh-ai-gateway-infra(58d1dfbc-e215-43af-9b9b-e9602ab9ea0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" Nov 01 09:45:15.051731 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:15.051680 5710 scope.go:122] "RemoveContainer" containerID="78c914bff14fb29af6644f40c9055b01918d919ca920544454677d871bd547de" Nov 01 09:45:15.482560 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:15.482515 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:45:15.589164 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:15.589123 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerStarted","Data":"e341de1b4c552e36a04821ff1673a05cd4baefa0e36bb0f7197208981c199194"} Nov 01 09:45:16.599774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:16.599712 5710 generic.go:356] "Generic (PLEG): container finished" podID="34951a7b-c05a-4e28-ad67-318a376b1616" containerID="e341de1b4c552e36a04821ff1673a05cd4baefa0e36bb0f7197208981c199194" exitCode=60 Nov 01 09:45:16.599774 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:16.599751 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerDied","Data":"e341de1b4c552e36a04821ff1673a05cd4baefa0e36bb0f7197208981c199194"} Nov 01 09:45:16.600313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:16.599822 5710 scope.go:122] "RemoveContainer" containerID="78c914bff14fb29af6644f40c9055b01918d919ca920544454677d871bd547de" Nov 01 09:45:16.600313 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:16.600247 5710 scope.go:122] "RemoveContainer" containerID="e341de1b4c552e36a04821ff1673a05cd4baefa0e36bb0f7197208981c199194" Nov 01 09:45:16.600629 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:16.600604 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8_odh-ai-gateway-infra(34951a7b-c05a-4e28-ad67-318a376b1616)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" Nov 01 09:45:16.709246 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:16.709204 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8"] Nov 01 09:45:17.055519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.055423 5710 scope.go:122] "RemoveContainer" containerID="681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d" Nov 01 09:45:17.055708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.055561 5710 scope.go:122] "RemoveContainer" containerID="0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10" Nov 01 09:45:17.613841 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.613798 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerStarted","Data":"e2eabfe1d4e4d225d1ca2337160d747ecc740fb4ccd28732858e9b53f6b858e7"} Nov 01 09:45:17.617997 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.617965 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerStarted","Data":"fd2ab4627cbb5911a156af16fa4ba617f41a6c84079b1cc4f2789fb25d35faa7"} Nov 01 09:45:17.685733 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.685633 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7"] Nov 01 09:45:17.815997 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.815969 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:17.872460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.872422 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5\" (UniqueName: \"kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5\") pod \"34951a7b-c05a-4e28-ad67-318a376b1616\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " Nov 01 09:45:17.872674 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.872477 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca\") pod \"34951a7b-c05a-4e28-ad67-318a376b1616\" (UID: \"34951a7b-c05a-4e28-ad67-318a376b1616\") " Nov 01 09:45:17.873377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.873336 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca" pod "34951a7b-c05a-4e28-ad67-318a376b1616" (UID: "34951a7b-c05a-4e28-ad67-318a376b1616"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:45:17.878020 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.875955 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5" pod "34951a7b-c05a-4e28-ad67-318a376b1616" (UID: "34951a7b-c05a-4e28-ad67-318a376b1616"). InnerVolumeSpecName "kube-api-access-p2hp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:17.974899 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.974815 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2hp5\" (UniqueName: \"kubernetes.io/projected/34951a7b-c05a-4e28-ad67-318a376b1616-kube-api-access-p2hp5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:17.974899 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:17.974851 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34951a7b-c05a-4e28-ad67-318a376b1616-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:18.051103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.051048 5710 scope.go:122] "RemoveContainer" containerID="b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4" Nov 01 09:45:18.628430 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.628394 5710 generic.go:356] "Generic (PLEG): container finished" podID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerID="e2eabfe1d4e4d225d1ca2337160d747ecc740fb4ccd28732858e9b53f6b858e7" exitCode=60 Nov 01 09:45:18.628962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.628471 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerDied","Data":"e2eabfe1d4e4d225d1ca2337160d747ecc740fb4ccd28732858e9b53f6b858e7"} Nov 01 09:45:18.628962 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.628569 5710 scope.go:122] "RemoveContainer" containerID="681a3837f8f85763467249c6d2dab95c92e0a5b2ee51ca93ab09a978aff6cf3d" Nov 01 09:45:18.629089 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.629023 5710 scope.go:122] "RemoveContainer" containerID="e2eabfe1d4e4d225d1ca2337160d747ecc740fb4ccd28732858e9b53f6b858e7" Nov 01 09:45:18.629421 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:18.629396 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw_odh-ai-gateway-infra(3b987afa-970c-4e07-93f2-a567ad2a7f2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" Nov 01 09:45:18.630225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.630203 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" Nov 01 09:45:18.630225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.630216 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8" event={"ID":"34951a7b-c05a-4e28-ad67-318a376b1616","Type":"ContainerDied","Data":"ed116dbf00d83aa3de8efa9afa750b6ca0707b30d84c7669ca2ab9707faed4b8"} Nov 01 09:45:18.632909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.632882 5710 generic.go:356] "Generic (PLEG): container finished" podID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerID="fd2ab4627cbb5911a156af16fa4ba617f41a6c84079b1cc4f2789fb25d35faa7" exitCode=60 Nov 01 09:45:18.633027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.632918 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerDied","Data":"fd2ab4627cbb5911a156af16fa4ba617f41a6c84079b1cc4f2789fb25d35faa7"} Nov 01 09:45:18.633374 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.633343 5710 scope.go:122] "RemoveContainer" containerID="fd2ab4627cbb5911a156af16fa4ba617f41a6c84079b1cc4f2789fb25d35faa7" Nov 01 09:45:18.633671 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:18.633651 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr_odh-ai-gateway-infra(58d1dfbc-e215-43af-9b9b-e9602ab9ea0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" Nov 01 09:45:18.635388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.635364 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerStarted","Data":"df0964eb0d05d939e5fdcd107df2c279b3f52a5afcd37587b95ed621504f4253"} Nov 01 09:45:18.643785 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.643767 5710 scope.go:122] "RemoveContainer" containerID="e341de1b4c552e36a04821ff1673a05cd4baefa0e36bb0f7197208981c199194" Nov 01 09:45:18.653531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.653508 5710 scope.go:122] "RemoveContainer" containerID="0946c65b5ac10b36fd3fa3321dc633ef8f3c87cb3f7db8fd3fb2afe196579d10" Nov 01 09:45:18.734159 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.734123 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw"] Nov 01 09:45:18.844006 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.843970 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr"] Nov 01 09:45:18.965222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.965187 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8"] Nov 01 09:45:18.975502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:18.975458 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a83714-29892105-lj9d8"] Nov 01 09:45:19.152229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.152132 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" path="/var/lib/kubelet/pods/34951a7b-c05a-4e28-ad67-318a376b1616/volumes" Nov 01 09:45:19.657147 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.657111 5710 generic.go:356] "Generic (PLEG): container finished" podID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerID="df0964eb0d05d939e5fdcd107df2c279b3f52a5afcd37587b95ed621504f4253" exitCode=60 Nov 01 09:45:19.657596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.657178 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerDied","Data":"df0964eb0d05d939e5fdcd107df2c279b3f52a5afcd37587b95ed621504f4253"} Nov 01 09:45:19.657596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.657238 5710 scope.go:122] "RemoveContainer" containerID="b0797a4c8de932d83dce95acb582dd802af30f013224e1cce26464c21c4c5fc4" Nov 01 09:45:19.657825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.657803 5710 scope.go:122] "RemoveContainer" containerID="df0964eb0d05d939e5fdcd107df2c279b3f52a5afcd37587b95ed621504f4253" Nov 01 09:45:19.658248 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:19.658214 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt_odh-ai-gateway-infra(47ca7164-be67-45b6-b5f2-b89535f0f70e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" Nov 01 09:45:19.843934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.843895 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:19.893124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.893090 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:19.947537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.947440 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj\" (UniqueName: \"kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj\") pod \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " Nov 01 09:45:19.947537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.947535 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca\") pod \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\" (UID: \"3b987afa-970c-4e07-93f2-a567ad2a7f2a\") " Nov 01 09:45:19.947983 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.947959 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca" pod "3b987afa-970c-4e07-93f2-a567ad2a7f2a" (UID: "3b987afa-970c-4e07-93f2-a567ad2a7f2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:45:19.949940 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:19.949918 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj" pod "3b987afa-970c-4e07-93f2-a567ad2a7f2a" (UID: "3b987afa-970c-4e07-93f2-a567ad2a7f2a"). InnerVolumeSpecName "kube-api-access-q5nlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:20.048640 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.048598 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca\") pod \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " Nov 01 09:45:20.048850 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.048817 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4\" (UniqueName: \"kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4\") pod \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\" (UID: \"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e\") " Nov 01 09:45:20.049039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.049015 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca" pod "58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" (UID: "58d1dfbc-e215-43af-9b9b-e9602ab9ea0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:45:20.049326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.049311 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:20.049371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.049330 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nlj\" (UniqueName: \"kubernetes.io/projected/3b987afa-970c-4e07-93f2-a567ad2a7f2a-kube-api-access-q5nlj\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:20.049371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.049341 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b987afa-970c-4e07-93f2-a567ad2a7f2a-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:20.051182 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.051153 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4" pod "58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" (UID: "58d1dfbc-e215-43af-9b9b-e9602ab9ea0e"). InnerVolumeSpecName "kube-api-access-dlld4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:20.120742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.116253 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt"] Nov 01 09:45:20.151623 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.151590 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlld4\" (UniqueName: \"kubernetes.io/projected/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e-kube-api-access-dlld4\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:20.682626 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.682593 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" event={"ID":"3b987afa-970c-4e07-93f2-a567ad2a7f2a","Type":"ContainerDied","Data":"b4272b73cb305e251e48fb6906e70c9a0f3c5b001dc974614b73b426c6394785"} Nov 01 09:45:20.683180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.682651 5710 scope.go:122] "RemoveContainer" containerID="e2eabfe1d4e4d225d1ca2337160d747ecc740fb4ccd28732858e9b53f6b858e7" Nov 01 09:45:20.683180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.682738 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw" Nov 01 09:45:20.691504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.690730 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" event={"ID":"58d1dfbc-e215-43af-9b9b-e9602ab9ea0e","Type":"ContainerDied","Data":"a1776b0bbe5cf39310b0a2737e3adb5803dae40cb46aeb31b6ac487dbe98d113"} Nov 01 09:45:20.691504 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.690755 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr" Nov 01 09:45:20.703275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.703248 5710 scope.go:122] "RemoveContainer" containerID="fd2ab4627cbb5911a156af16fa4ba617f41a6c84079b1cc4f2789fb25d35faa7" Nov 01 09:45:20.853119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.853073 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr"] Nov 01 09:45:20.876601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.876562 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:20.898373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.898320 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b711ae-29892105-gfgmr"] Nov 01 09:45:20.981858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.981761 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca\") pod \"47ca7164-be67-45b6-b5f2-b89535f0f70e\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " Nov 01 09:45:20.981858 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.981820 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk\" (UniqueName: \"kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk\") pod \"47ca7164-be67-45b6-b5f2-b89535f0f70e\" (UID: \"47ca7164-be67-45b6-b5f2-b89535f0f70e\") " Nov 01 09:45:20.982185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.982158 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca" pod "47ca7164-be67-45b6-b5f2-b89535f0f70e" (UID: "47ca7164-be67-45b6-b5f2-b89535f0f70e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:45:20.982315 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.982299 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca7164-be67-45b6-b5f2-b89535f0f70e-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:20.984321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:20.984291 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk" pod "47ca7164-be67-45b6-b5f2-b89535f0f70e" (UID: "47ca7164-be67-45b6-b5f2-b89535f0f70e"). InnerVolumeSpecName "kube-api-access-pstnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:21.060000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.059965 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" path="/var/lib/kubelet/pods/58d1dfbc-e215-43af-9b9b-e9602ab9ea0e/volumes" Nov 01 09:45:21.083507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.083467 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstnk\" (UniqueName: \"kubernetes.io/projected/47ca7164-be67-45b6-b5f2-b89535f0f70e-kube-api-access-pstnk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:21.111529 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.111492 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw"] Nov 01 09:45:21.122059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.122010 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f41050-29892105-w82gw"] Nov 01 09:45:21.707420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.707386 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" Nov 01 09:45:21.708026 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.707395 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt" event={"ID":"47ca7164-be67-45b6-b5f2-b89535f0f70e","Type":"ContainerDied","Data":"12393f8e59be345bc29817c3ddabb17114e041737a19442efe0a5834187121a2"} Nov 01 09:45:21.708026 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.707521 5710 scope.go:122] "RemoveContainer" containerID="df0964eb0d05d939e5fdcd107df2c279b3f52a5afcd37587b95ed621504f4253" Nov 01 09:45:21.900765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.892302 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt"] Nov 01 09:45:21.900765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:21.900598 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-343207-29892105-s4bbt"] Nov 01 09:45:22.686335 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:22.686292 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" podUID="c773e000-899a-44e0-bbb5-2014e54132df" containerName="payload-pre-processing" containerID="cri-o://396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20" gracePeriod=25 Nov 01 09:45:22.969219 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:22.969190 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:45:23.051993 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.051963 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume\") pod \"c773e000-899a-44e0-bbb5-2014e54132df\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " Nov 01 09:45:23.052203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.052094 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp\" (UniqueName: \"kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp\") pod \"c773e000-899a-44e0-bbb5-2014e54132df\" (UID: \"c773e000-899a-44e0-bbb5-2014e54132df\") " Nov 01 09:45:23.052348 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.052327 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume" pod "c773e000-899a-44e0-bbb5-2014e54132df" (UID: "c773e000-899a-44e0-bbb5-2014e54132df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:45:23.052559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.052543 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c773e000-899a-44e0-bbb5-2014e54132df-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:23.054639 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.054610 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp" pod "c773e000-899a-44e0-bbb5-2014e54132df" (UID: "c773e000-899a-44e0-bbb5-2014e54132df"). InnerVolumeSpecName "kube-api-access-msrfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:23.060209 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.060183 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" path="/var/lib/kubelet/pods/3b987afa-970c-4e07-93f2-a567ad2a7f2a/volumes" Nov 01 09:45:23.060650 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.060636 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" path="/var/lib/kubelet/pods/47ca7164-be67-45b6-b5f2-b89535f0f70e/volumes" Nov 01 09:45:23.154332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.154231 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrfp\" (UniqueName: \"kubernetes.io/projected/c773e000-899a-44e0-bbb5-2014e54132df-kube-api-access-msrfp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:23.726455 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.726413 5710 generic.go:356] "Generic (PLEG): container finished" podID="c773e000-899a-44e0-bbb5-2014e54132df" containerID="396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20" exitCode=0 Nov 01 09:45:23.726683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.726482 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" Nov 01 09:45:23.726683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.726503 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" event={"ID":"c773e000-899a-44e0-bbb5-2014e54132df","Type":"ContainerDied","Data":"396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20"} Nov 01 09:45:23.726683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.726607 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7" event={"ID":"c773e000-899a-44e0-bbb5-2014e54132df","Type":"ContainerDied","Data":"9fddf243ee7b95c8e5b339639028fbb7d1805d166ebe688eac9308dc25e3a873"} Nov 01 09:45:23.726683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.726640 5710 scope.go:122] "RemoveContainer" containerID="396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20" Nov 01 09:45:23.741468 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.741432 5710 scope.go:122] "RemoveContainer" containerID="396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20" Nov 01 09:45:23.742041 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:23.742012 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20\": container with ID starting with 396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20 not found: ID does not exist" containerID="396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20" Nov 01 09:45:23.742138 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.742049 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20"} err="failed to get container status \"396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20\": rpc error: code = NotFound desc = could not find container \"396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20\": container with ID starting with 396bf52f58a8cb525e09018b6e454b765bb9cb7124c4bd913fdfc814fb4f8a20 not found: ID does not exist" Nov 01 09:45:23.880539 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.880500 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7"] Nov 01 09:45:23.886558 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:23.886523 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-12a01a-6d867dd69f-2knb7"] Nov 01 09:45:24.639897 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:24.639864 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:45:25.060818 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:25.060730 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c773e000-899a-44e0-bbb5-2014e54132df" path="/var/lib/kubelet/pods/c773e000-899a-44e0-bbb5-2014e54132df/volumes" Nov 01 09:45:45.479438 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:45.479393 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:45:48.100655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:48.100566 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf"] Nov 01 09:45:48.101301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:48.100873 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" containerID="cri-o://bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df" gracePeriod=30 Nov 01 09:45:52.925915 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:52.925870 5710 patch_prober.go:29] interesting pod/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 09:45:52.926417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:52.925943 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 09:45:53.400257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.400227 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:45:53.538463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538379 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538421 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538474 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538502 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538582 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538616 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538662 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.538744 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538685 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.539064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538735 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:45:53.539064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538775 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data\") pod \"182eb899-9d31-4514-b545-de771d99bc4b\" (UID: \"182eb899-9d31-4514-b545-de771d99bc4b\") " Nov 01 09:45:53.539064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538944 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:45:53.539064 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.538964 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:45:53.539285 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.539244 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:45:53.539574 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.539549 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:45:53.539634 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.539616 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.539681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.539643 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.539681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.539659 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.539681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.539672 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.541054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.541031 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:45:53.541394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.541376 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:53.541506 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.541491 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:45:53.541693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.541679 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f" pod "182eb899-9d31-4514-b545-de771d99bc4b" (UID: "182eb899-9d31-4514-b545-de771d99bc4b"). InnerVolumeSpecName "kube-api-access-p574f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:45:53.640666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.640632 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182eb899-9d31-4514-b545-de771d99bc4b-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.640666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.640663 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182eb899-9d31-4514-b545-de771d99bc4b-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.640918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.640679 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p574f\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-kube-api-access-p574f\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.640918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.640694 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182eb899-9d31-4514-b545-de771d99bc4b-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.640918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.640703 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182eb899-9d31-4514-b545-de771d99bc4b-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:45:53.963011 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.962977 5710 generic.go:356] "Generic (PLEG): container finished" podID="182eb899-9d31-4514-b545-de771d99bc4b" containerID="bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df" exitCode=0 Nov 01 09:45:53.963490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.963035 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" Nov 01 09:45:53.963490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.963056 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" event={"ID":"182eb899-9d31-4514-b545-de771d99bc4b","Type":"ContainerDied","Data":"bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df"} Nov 01 09:45:53.963490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.963123 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf" event={"ID":"182eb899-9d31-4514-b545-de771d99bc4b","Type":"ContainerDied","Data":"38c65aeb0a83e27722a7b4f262d94bb9573b9b243b9afb07d1f66f9424e9a907"} Nov 01 09:45:53.963490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.963155 5710 scope.go:122] "RemoveContainer" containerID="bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df" Nov 01 09:45:53.975088 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.975062 5710 scope.go:122] "RemoveContainer" containerID="725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14" Nov 01 09:45:53.985925 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.985898 5710 scope.go:122] "RemoveContainer" containerID="bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df" Nov 01 09:45:53.986291 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:53.986270 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df\": container with ID starting with bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df not found: ID does not exist" containerID="bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df" Nov 01 09:45:53.986367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.986303 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df"} err="failed to get container status \"bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df\": rpc error: code = NotFound desc = could not find container \"bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df\": container with ID starting with bf00dca1ade40cf4d2950e96b593d7fcf84210d234bac79527238fab566119df not found: ID does not exist" Nov 01 09:45:53.986367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.986323 5710 scope.go:122] "RemoveContainer" containerID="725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14" Nov 01 09:45:53.986629 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:45:53.986595 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14\": container with ID starting with 725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14 not found: ID does not exist" containerID="725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14" Nov 01 09:45:53.986693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:53.986636 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14"} err="failed to get container status \"725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14\": rpc error: code = NotFound desc = could not find container \"725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14\": container with ID starting with 725eb62bf8d429ca0e0e307b7f8b8647b63ee2efaeadb1093d092df7e50a5f14 not found: ID does not exist" Nov 01 09:45:54.118271 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:54.118236 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf"] Nov 01 09:45:54.164201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:54.164167 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49f18f-openshift-default-b59794bd7-ht8cf"] Nov 01 09:45:55.059985 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:45:55.059948 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182eb899-9d31-4514-b545-de771d99bc4b" path="/var/lib/kubelet/pods/182eb899-9d31-4514-b545-de771d99bc4b/volumes" Nov 01 09:46:10.871932 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:46:10.871882 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vbndd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:46:10.872471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:46:10.871965 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:46:15.481973 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:46:15.481924 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:46:37.067991 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:46:37.067905 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:46:45.478100 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:46:45.478057 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:47:15.503554 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:47:15.503513 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:47:41.893691 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:47:41.893650 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:47:45.481037 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:47:45.480992 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:47:56.761488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:47:56.761446 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n"] Nov 01 09:48:01.285298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:01.285265 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:48:01.298276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:01.298250 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:48:01.763003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:01.762943 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" podUID="b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" containerName="payload-pre-processing" containerID="cri-o://6194ea206f2c41c02055cbd5c736003772eb0ef733ff51842dc3eae55140b96e" gracePeriod=25 Nov 01 09:48:01.969910 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:01.969875 5710 generic.go:356] "Generic (PLEG): container finished" podID="b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" containerID="6194ea206f2c41c02055cbd5c736003772eb0ef733ff51842dc3eae55140b96e" exitCode=0 Nov 01 09:48:01.970117 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:01.969946 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" event={"ID":"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd","Type":"ContainerDied","Data":"6194ea206f2c41c02055cbd5c736003772eb0ef733ff51842dc3eae55140b96e"} Nov 01 09:48:02.053938 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.053907 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:48:02.240560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.240518 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume\") pod \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " Nov 01 09:48:02.240826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.240811 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2\" (UniqueName: \"kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2\") pod \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\" (UID: \"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd\") " Nov 01 09:48:02.241000 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.240968 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume" pod "b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" (UID: "b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:48:02.241303 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.241282 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:02.243174 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.243136 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2" pod "b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" (UID: "b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd"). InnerVolumeSpecName "kube-api-access-s77c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:48:02.343339 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.343302 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s77c2\" (UniqueName: \"kubernetes.io/projected/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd-kube-api-access-s77c2\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:02.978102 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.978071 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" Nov 01 09:48:02.978291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.978108 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n" event={"ID":"b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd","Type":"ContainerDied","Data":"304a24d8f09f95ea42b4a2b7f4dda9f7cc240933afcfe65cb8390414c21cd063"} Nov 01 09:48:02.978291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:02.978172 5710 scope.go:122] "RemoveContainer" containerID="6194ea206f2c41c02055cbd5c736003772eb0ef733ff51842dc3eae55140b96e" Nov 01 09:48:03.132870 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:03.132829 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n"] Nov 01 09:48:03.179203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:03.179168 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3d6bb-5bbfb46f95-pgv6n"] Nov 01 09:48:05.060436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:05.060361 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" path="/var/lib/kubelet/pods/b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd/volumes" Nov 01 09:48:15.482912 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:48:15.482869 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:48:23.586900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:23.586851 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7"] Nov 01 09:48:23.700369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:23.700332 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z"] Nov 01 09:48:28.588177 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:28.588128 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" podUID="56390f5e-1626-437c-bbc8-4248040edacb" containerName="payload-processing" containerID="cri-o://2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1" gracePeriod=25 Nov 01 09:48:28.700896 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:28.700830 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" podUID="54a3c8fe-6345-4675-8883-46208026215c" containerName="payload-pre-processing" containerID="cri-o://38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8" gracePeriod=25 Nov 01 09:48:28.864792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:28.864760 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:48:28.984842 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:28.984808 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:48:29.021225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.021180 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume\") pod \"56390f5e-1626-437c-bbc8-4248040edacb\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " Nov 01 09:48:29.021464 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.021292 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w\" (UniqueName: \"kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w\") pod \"56390f5e-1626-437c-bbc8-4248040edacb\" (UID: \"56390f5e-1626-437c-bbc8-4248040edacb\") " Nov 01 09:48:29.021705 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.021669 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume" pod "56390f5e-1626-437c-bbc8-4248040edacb" (UID: "56390f5e-1626-437c-bbc8-4248040edacb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:48:29.021933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.021910 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56390f5e-1626-437c-bbc8-4248040edacb-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.024170 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.024137 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w" pod "56390f5e-1626-437c-bbc8-4248040edacb" (UID: "56390f5e-1626-437c-bbc8-4248040edacb"). InnerVolumeSpecName "kube-api-access-qwv9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:48:29.123097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.122999 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2\" (UniqueName: \"kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2\") pod \"54a3c8fe-6345-4675-8883-46208026215c\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " Nov 01 09:48:29.123305 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.123114 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume\") pod \"54a3c8fe-6345-4675-8883-46208026215c\" (UID: \"54a3c8fe-6345-4675-8883-46208026215c\") " Nov 01 09:48:29.123562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.123536 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume" pod "54a3c8fe-6345-4675-8883-46208026215c" (UID: "54a3c8fe-6345-4675-8883-46208026215c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:48:29.123649 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.123638 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwv9w\" (UniqueName: \"kubernetes.io/projected/56390f5e-1626-437c-bbc8-4248040edacb-kube-api-access-qwv9w\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.123740 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.123655 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a3c8fe-6345-4675-8883-46208026215c-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.125569 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.125536 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2" pod "54a3c8fe-6345-4675-8883-46208026215c" (UID: "54a3c8fe-6345-4675-8883-46208026215c"). InnerVolumeSpecName "kube-api-access-4hst2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:48:29.150603 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.150560 5710 generic.go:356] "Generic (PLEG): container finished" podID="56390f5e-1626-437c-bbc8-4248040edacb" containerID="2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1" exitCode=0 Nov 01 09:48:29.150825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.150608 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" Nov 01 09:48:29.150825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.150652 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" event={"ID":"56390f5e-1626-437c-bbc8-4248040edacb","Type":"ContainerDied","Data":"2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1"} Nov 01 09:48:29.150825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.150758 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7" event={"ID":"56390f5e-1626-437c-bbc8-4248040edacb","Type":"ContainerDied","Data":"8c5e70b62e08aef37d4d5a8873dcee60a64d8d0fc7dfe458e20d6ab061383f57"} Nov 01 09:48:29.150825 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.150794 5710 scope.go:122] "RemoveContainer" containerID="2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1" Nov 01 09:48:29.152362 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.152338 5710 generic.go:356] "Generic (PLEG): container finished" podID="54a3c8fe-6345-4675-8883-46208026215c" containerID="38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8" exitCode=0 Nov 01 09:48:29.152469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.152403 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" event={"ID":"54a3c8fe-6345-4675-8883-46208026215c","Type":"ContainerDied","Data":"38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8"} Nov 01 09:48:29.152469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.152441 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" event={"ID":"54a3c8fe-6345-4675-8883-46208026215c","Type":"ContainerDied","Data":"063af15987fcc428fc9e150d7f00514e0696ecdd213005413c6c1f0dce949519"} Nov 01 09:48:29.152588 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.152543 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z" Nov 01 09:48:29.166343 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.166321 5710 scope.go:122] "RemoveContainer" containerID="2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1" Nov 01 09:48:29.166650 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:48:29.166626 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1\": container with ID starting with 2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1 not found: ID does not exist" containerID="2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1" Nov 01 09:48:29.166708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.166659 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1"} err="failed to get container status \"2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1\": rpc error: code = NotFound desc = could not find container \"2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1\": container with ID starting with 2dd4830accb5fd4051086ab9575ba08d1afb9770c067825b414c340f8f68e1c1 not found: ID does not exist" Nov 01 09:48:29.166708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.166688 5710 scope.go:122] "RemoveContainer" containerID="38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8" Nov 01 09:48:29.180331 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.180305 5710 scope.go:122] "RemoveContainer" containerID="38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8" Nov 01 09:48:29.180755 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:48:29.180699 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8\": container with ID starting with 38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8 not found: ID does not exist" containerID="38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8" Nov 01 09:48:29.180874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.180762 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8"} err="failed to get container status \"38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8\": rpc error: code = NotFound desc = could not find container \"38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8\": container with ID starting with 38017d1806255f8f76178c626ade2dd8f562a625a34e55ee72e9f3f5430c38a8 not found: ID does not exist" Nov 01 09:48:29.225211 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.225169 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hst2\" (UniqueName: \"kubernetes.io/projected/54a3c8fe-6345-4675-8883-46208026215c-kube-api-access-4hst2\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.305160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.305124 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z"] Nov 01 09:48:29.311180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.311137 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh"] Nov 01 09:48:29.311445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.311410 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" podUID="4336fec2-f181-4506-bca4-283fa60af703" containerName="maas-api" containerID="cri-o://33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d" gracePeriod=30 Nov 01 09:48:29.314224 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.314196 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f41050-5bd78b8bcf-zjw2z"] Nov 01 09:48:29.415229 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.415142 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7"] Nov 01 09:48:29.419275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.419245 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f41050-785765bbff-pbqh7"] Nov 01 09:48:29.648666 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.648634 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:48:29.707800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.703292 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72\" (UniqueName: \"kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72\") pod \"4336fec2-f181-4506-bca4-283fa60af703\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " Nov 01 09:48:29.707800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.703401 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs\") pod \"4336fec2-f181-4506-bca4-283fa60af703\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " Nov 01 09:48:29.707800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.703455 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\") pod \"4336fec2-f181-4506-bca4-283fa60af703\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " Nov 01 09:48:29.707800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.703526 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca\") pod \"4336fec2-f181-4506-bca4-283fa60af703\" (UID: \"4336fec2-f181-4506-bca4-283fa60af703\") " Nov 01 09:48:29.707800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.704493 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca" pod "4336fec2-f181-4506-bca4-283fa60af703" (UID: "4336fec2-f181-4506-bca4-283fa60af703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:48:29.714144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.710444 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72" pod "4336fec2-f181-4506-bca4-283fa60af703" (UID: "4336fec2-f181-4506-bca4-283fa60af703"). InnerVolumeSpecName "kube-api-access-2cz72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:48:29.714144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.711257 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs" pod "4336fec2-f181-4506-bca4-283fa60af703" (UID: "4336fec2-f181-4506-bca4-283fa60af703"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:48:29.753769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.753710 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls" pod "4336fec2-f181-4506-bca4-283fa60af703" (UID: "4336fec2-f181-4506-bca4-283fa60af703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:48:29.805144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.805103 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.805144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.805135 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4336fec2-f181-4506-bca4-283fa60af703-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.805144 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.805145 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4336fec2-f181-4506-bca4-283fa60af703-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:29.805396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:29.805166 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cz72\" (UniqueName: \"kubernetes.io/projected/4336fec2-f181-4506-bca4-283fa60af703-kube-api-access-2cz72\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:48:30.161500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.161458 5710 generic.go:356] "Generic (PLEG): container finished" podID="4336fec2-f181-4506-bca4-283fa60af703" containerID="33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d" exitCode=0 Nov 01 09:48:30.161803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.161515 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" Nov 01 09:48:30.161803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.161553 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" event={"ID":"4336fec2-f181-4506-bca4-283fa60af703","Type":"ContainerDied","Data":"33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d"} Nov 01 09:48:30.161803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.161625 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh" event={"ID":"4336fec2-f181-4506-bca4-283fa60af703","Type":"ContainerDied","Data":"c0856905aab5f7a0394172886c4fdc12ba479b01b29e9d02a93ba299e82d6333"} Nov 01 09:48:30.161803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.161662 5710 scope.go:122] "RemoveContainer" containerID="33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d" Nov 01 09:48:30.173934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.173902 5710 scope.go:122] "RemoveContainer" containerID="33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d" Nov 01 09:48:30.174285 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:48:30.174265 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d\": container with ID starting with 33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d not found: ID does not exist" containerID="33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d" Nov 01 09:48:30.174378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.174294 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d"} err="failed to get container status \"33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d\": rpc error: code = NotFound desc = could not find container \"33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d\": container with ID starting with 33c12304b758cf23316573bde4b0227d5c4c3110618c283cbb9298b6324ff72d not found: ID does not exist" Nov 01 09:48:30.335789 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.333352 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh"] Nov 01 09:48:30.341521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:30.341487 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f41050-8584d64769-569bh"] Nov 01 09:48:31.059424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:31.059342 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4336fec2-f181-4506-bca4-283fa60af703" path="/var/lib/kubelet/pods/4336fec2-f181-4506-bca4-283fa60af703/volumes" Nov 01 09:48:31.059859 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:31.059794 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a3c8fe-6345-4675-8883-46208026215c" path="/var/lib/kubelet/pods/54a3c8fe-6345-4675-8883-46208026215c/volumes" Nov 01 09:48:31.060150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:31.060136 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56390f5e-1626-437c-bbc8-4248040edacb" path="/var/lib/kubelet/pods/56390f5e-1626-437c-bbc8-4248040edacb/volumes" Nov 01 09:48:32.895350 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:48:32.895308 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:48:45.478597 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:48:45.478551 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:49:03.937790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:03.937710 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2"] Nov 01 09:49:03.938348 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:03.938048 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" containerID="cri-o://4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214" gracePeriod=30 Nov 01 09:49:09.227298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.227265 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:49:09.263596 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263557 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.263811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263612 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.263811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263643 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.263811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263665 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.263811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263687 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.264012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263867 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.264012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263902 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.264012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263942 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:09.264012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.263960 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.264234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264037 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy\") pod \"a53917f1-1232-480e-b014-e81651753c9a\" (UID: \"a53917f1-1232-480e-b014-e81651753c9a\") " Nov 01 09:49:09.264234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264149 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:09.264234 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264172 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:09.264388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264242 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:09.264440 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264381 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:49:09.264752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264615 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.264752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264641 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.264752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264654 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.264752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264663 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53917f1-1232-480e-b014-e81651753c9a-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.264752 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.264674 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.266296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.266269 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "kube-api-access-dvt6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:49:09.266424 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.266408 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:49:09.266833 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.266812 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:09.266953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.266815 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token" pod "a53917f1-1232-480e-b014-e81651753c9a" (UID: "a53917f1-1232-480e-b014-e81651753c9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:49:09.365456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.365416 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53917f1-1232-480e-b014-e81651753c9a-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.365456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.365449 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvt6n\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-kube-api-access-dvt6n\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.365456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.365460 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53917f1-1232-480e-b014-e81651753c9a-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.365713 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.365470 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53917f1-1232-480e-b014-e81651753c9a-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:09.415972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.415931 5710 generic.go:356] "Generic (PLEG): container finished" podID="a53917f1-1232-480e-b014-e81651753c9a" containerID="4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214" exitCode=0 Nov 01 09:49:09.416196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.416004 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" Nov 01 09:49:09.416196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.416067 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" event={"ID":"a53917f1-1232-480e-b014-e81651753c9a","Type":"ContainerDied","Data":"4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214"} Nov 01 09:49:09.416196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.416124 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2" event={"ID":"a53917f1-1232-480e-b014-e81651753c9a","Type":"ContainerDied","Data":"137c87069dabac8cb32fcfd02addae0f33d96b314ddddffe1808a472521763a4"} Nov 01 09:49:09.416196 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.416152 5710 scope.go:122] "RemoveContainer" containerID="4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214" Nov 01 09:49:09.431297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.431216 5710 scope.go:122] "RemoveContainer" containerID="19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3" Nov 01 09:49:09.441912 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.441885 5710 scope.go:122] "RemoveContainer" containerID="4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214" Nov 01 09:49:09.442227 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:49:09.442197 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214\": container with ID starting with 4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214 not found: ID does not exist" containerID="4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214" Nov 01 09:49:09.442301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.442239 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214"} err="failed to get container status \"4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214\": rpc error: code = NotFound desc = could not find container \"4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214\": container with ID starting with 4646218a445af73a1893c8cc48263ad9e64f2c13652584282341a1ba444c7214 not found: ID does not exist" Nov 01 09:49:09.442301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.442257 5710 scope.go:122] "RemoveContainer" containerID="19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3" Nov 01 09:49:09.442528 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:49:09.442508 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3\": container with ID starting with 19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3 not found: ID does not exist" containerID="19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3" Nov 01 09:49:09.442582 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.442537 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3"} err="failed to get container status \"19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3\": rpc error: code = NotFound desc = could not find container \"19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3\": container with ID starting with 19ff66bd22bda853c3409658b5e86b06e94e6e2ab8685d78c526ceeb04d400d3 not found: ID does not exist" Nov 01 09:49:09.571171 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.571082 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2"] Nov 01 09:49:09.575549 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:09.575514 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c81a9d-openshift-default-d98659877-4rhx2"] Nov 01 09:49:11.061053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:11.061018 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53917f1-1232-480e-b014-e81651753c9a" path="/var/lib/kubelet/pods/a53917f1-1232-480e-b014-e81651753c9a/volumes" Nov 01 09:49:15.477654 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:49:15.477615 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:49:20.140894 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:20.140845 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn"] Nov 01 09:49:20.141494 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:20.141146 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" containerID="cri-o://0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a" gracePeriod=30 Nov 01 09:49:25.455090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.455059 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:49:25.542284 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.542250 5710 generic.go:356] "Generic (PLEG): container finished" podID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerID="0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a" exitCode=0 Nov 01 09:49:25.542507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.542305 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" Nov 01 09:49:25.542507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.542333 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" event={"ID":"8622048f-3e2c-462d-bbd7-b1359d75a387","Type":"ContainerDied","Data":"0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a"} Nov 01 09:49:25.542507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.542394 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn" event={"ID":"8622048f-3e2c-462d-bbd7-b1359d75a387","Type":"ContainerDied","Data":"21bc061e666276b0ea19c9e5a060c499e541c618be4e3eb472e94880c03678dc"} Nov 01 09:49:25.542507 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.542420 5710 scope.go:122] "RemoveContainer" containerID="0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a" Nov 01 09:49:25.553319 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.553301 5710 scope.go:122] "RemoveContainer" containerID="8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a" Nov 01 09:49:25.563573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.563549 5710 scope.go:122] "RemoveContainer" containerID="0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a" Nov 01 09:49:25.563984 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:49:25.563961 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a\": container with ID starting with 0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a not found: ID does not exist" containerID="0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a" Nov 01 09:49:25.564051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.563994 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a"} err="failed to get container status \"0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a\": rpc error: code = NotFound desc = could not find container \"0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a\": container with ID starting with 0c3c7a75b0e32ea428fe4db65f0cf2dc35db8af5d13c01691802981e3c377f6a not found: ID does not exist" Nov 01 09:49:25.564051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.564011 5710 scope.go:122] "RemoveContainer" containerID="8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a" Nov 01 09:49:25.564260 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:49:25.564241 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a\": container with ID starting with 8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a not found: ID does not exist" containerID="8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a" Nov 01 09:49:25.564301 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.564267 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a"} err="failed to get container status \"8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a\": rpc error: code = NotFound desc = could not find container \"8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a\": container with ID starting with 8889815cdf282afeaafba7873e8e5512bef58cf9ed91633780dfa08376ef783a not found: ID does not exist" Nov 01 09:49:25.586933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.586902 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.586949 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587039 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587168 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587237 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587216 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587240 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:25.587332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587281 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587326 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587385 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587389 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:25.587454 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587414 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:25.587620 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587480 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket\") pod \"8622048f-3e2c-462d-bbd7-b1359d75a387\" (UID: \"8622048f-3e2c-462d-bbd7-b1359d75a387\") " Nov 01 09:49:25.587761 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587696 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:49:25.587958 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.587936 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:25.588371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.588347 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.588462 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.588379 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.588462 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.588395 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.588462 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.588408 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.588462 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.588420 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622048f-3e2c-462d-bbd7-b1359d75a387-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.589534 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.589508 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:49:25.589777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.589752 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:49:25.589777 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.589764 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:49:25.590007 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.589988 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6" pod "8622048f-3e2c-462d-bbd7-b1359d75a387" (UID: "8622048f-3e2c-462d-bbd7-b1359d75a387"). InnerVolumeSpecName "kube-api-access-k46d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:49:25.689561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.689472 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46d6\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-kube-api-access-k46d6\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.689561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.689505 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.689561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.689515 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.689561 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.689524 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622048f-3e2c-462d-bbd7-b1359d75a387-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:49:25.957703 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.957611 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn"] Nov 01 09:49:25.961227 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:25.961197 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b3d6bb-openshift-default-794b8f7d55-5gnfn"] Nov 01 09:49:27.059957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:27.059918 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" path="/var/lib/kubelet/pods/8622048f-3e2c-462d-bbd7-b1359d75a387/volumes" Nov 01 09:49:35.279262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:35.279159 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:49:40.163003 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:40.162959 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rjfq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 09:49:40.163707 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:49:40.163652 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 09:49:45.479995 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:49:45.479915 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:50:15.479951 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:50:15.479906 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:50:30.966371 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:50:30.966328 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:50:45.479267 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:50:45.479230 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:51:15.480143 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:51:15.480092 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:51:45.477807 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:51:45.477771 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:51:50.702531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:51:50.702480 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:52:15.480270 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:52:15.480228 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:52:40.658007 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.657907 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b"] Nov 01 09:52:40.658655 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658639 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658657 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658671 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" containerName="payload-pre-processing" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658676 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" containerName="payload-pre-processing" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658691 5710 cpu_manager.go:414] "containerMap: removing container" podUID="56390f5e-1626-437c-bbc8-4248040edacb" containerName="payload-processing" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658696 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56390f5e-1626-437c-bbc8-4248040edacb" containerName="payload-processing" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658703 5710 cpu_manager.go:414] "containerMap: removing container" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.658709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658707 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658729 5710 cpu_manager.go:414] "containerMap: removing container" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658735 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658745 5710 cpu_manager.go:414] "containerMap: removing container" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658749 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658757 5710 cpu_manager.go:414] "containerMap: removing container" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658761 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658775 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c773e000-899a-44e0-bbb5-2014e54132df" containerName="payload-pre-processing" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658780 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c773e000-899a-44e0-bbb5-2014e54132df" containerName="payload-pre-processing" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658791 5710 cpu_manager.go:414] "containerMap: removing container" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658795 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658802 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658808 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658820 5710 cpu_manager.go:414] "containerMap: removing container" podUID="4336fec2-f181-4506-bca4-283fa60af703" containerName="maas-api" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658824 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4336fec2-f181-4506-bca4-283fa60af703" containerName="maas-api" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658832 5710 cpu_manager.go:414] "containerMap: removing container" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658836 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658842 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658846 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658855 5710 cpu_manager.go:414] "containerMap: removing container" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658860 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658870 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658874 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658882 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658886 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658893 5710 cpu_manager.go:414] "containerMap: removing container" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658897 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658902 5710 cpu_manager.go:414] "containerMap: removing container" podUID="54a3c8fe-6345-4675-8883-46208026215c" containerName="payload-pre-processing" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658907 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a3c8fe-6345-4675-8883-46208026215c" containerName="payload-pre-processing" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658915 5710 cpu_manager.go:414] "containerMap: removing container" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658920 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658933 5710 cpu_manager.go:414] "containerMap: removing container" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:52:40.658974 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.658937 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659042 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a3c8fe-6345-4675-8883-46208026215c" containerName="payload-pre-processing" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659050 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659057 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659064 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659069 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659074 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659080 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1277329-4de9-4ba2-bea4-f0c9d2cc5cdd" containerName="payload-pre-processing" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659085 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="56390f5e-1626-437c-bbc8-4248040edacb" containerName="payload-processing" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659090 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659095 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659100 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659106 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c773e000-899a-44e0-bbb5-2014e54132df" containerName="payload-pre-processing" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659113 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659117 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659124 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="4336fec2-f181-4506-bca4-283fa60af703" containerName="maas-api" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659129 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d1dfbc-e215-43af-9b9b-e9602ab9ea0e" containerName="cleanup" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659135 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659140 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8622048f-3e2c-462d-bbd7-b1359d75a387" containerName="istio-proxy" Nov 01 09:52:40.660017 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.659144 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:52:40.662593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.662576 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.722363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.722324 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b"] Nov 01 09:52:40.765743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.759748 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.765743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.759864 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnpz\" (UniqueName: \"kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz\") pod \"payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.772129 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.772092 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0401bd23-dockercfg-bdzrw\"" Nov 01 09:52:40.772341 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.772185 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0401bd23\"" Nov 01 09:52:40.860953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.860908 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.861152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.860983 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnpz\" (UniqueName: \"kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz\") pod \"payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.861934 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.861910 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.936109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.936024 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnpz\" (UniqueName: \"kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz\") pod \"payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:40.977894 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:40.977852 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:41.378011 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:41.377980 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b"] Nov 01 09:52:41.380523 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:52:41.380485 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5512ce5_e232_4b9b_b8f5_19b0300f2f07.slice/crio-09c1b7ae53f22052f56babf9feca8e4bc6c2438983d6020048c3c15818183578 WatchSource:0}: Error finding container 09c1b7ae53f22052f56babf9feca8e4bc6c2438983d6020048c3c15818183578: Status 404 returned error can't find the container with id 09c1b7ae53f22052f56babf9feca8e4bc6c2438983d6020048c3c15818183578 Nov 01 09:52:41.996514 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:41.996477 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" event={"ID":"b5512ce5-e232-4b9b-b8f5-19b0300f2f07","Type":"ContainerStarted","Data":"c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8"} Nov 01 09:52:41.997042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:41.996531 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" event={"ID":"b5512ce5-e232-4b9b-b8f5-19b0300f2f07","Type":"ContainerStarted","Data":"09c1b7ae53f22052f56babf9feca8e4bc6c2438983d6020048c3c15818183578"} Nov 01 09:52:41.997042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:41.996592 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:42.117515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:42.117458 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" podStartSLOduration=2.117442246 podStartE2EDuration="2.117442246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:52:42.11485092 +0000 UTC m=+4785.203187605" watchObservedRunningTime="2026-11-01 09:52:42.117442246 +0000 UTC m=+4785.205778918" Nov 01 09:52:44.095425 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:44.095389 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8"] Nov 01 09:52:44.193552 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:44.193515 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5"] Nov 01 09:52:45.480873 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:52:45.480834 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:52:49.096601 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:49.096547 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerName="payload-processing" containerID="cri-o://79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772" gracePeriod=25 Nov 01 09:52:49.194804 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:49.194750 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerName="payload-pre-processing" containerID="cri-o://3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309" gracePeriod=25 Nov 01 09:52:53.005673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:53.005642 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:52:56.295053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:56.295008 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:52:57.306409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:52:57.306364 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:53:01.334057 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:01.334024 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:53:01.344189 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:01.344160 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:53:06.295158 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:06.295116 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:53:07.306453 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:07.306413 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:53:09.373067 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.372988 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:53:09.402600 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.402571 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume\") pod \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " Nov 01 09:53:09.402797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.402761 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l\" (UniqueName: \"kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l\") pod \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\" (UID: \"919c821a-75eb-4d38-8722-5e7d19d5fd8d\") " Nov 01 09:53:09.403009 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.402983 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume" pod "919c821a-75eb-4d38-8722-5e7d19d5fd8d" (UID: "919c821a-75eb-4d38-8722-5e7d19d5fd8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:53:09.403184 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.403164 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919c821a-75eb-4d38-8722-5e7d19d5fd8d-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:09.415757 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.405889 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l" pod "919c821a-75eb-4d38-8722-5e7d19d5fd8d" (UID: "919c821a-75eb-4d38-8722-5e7d19d5fd8d"). InnerVolumeSpecName "kube-api-access-5dv5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:53:09.461848 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.461817 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:53:09.504049 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.504010 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume\") pod \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " Nov 01 09:53:09.504269 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.504249 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82\" (UniqueName: \"kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82\") pod \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\" (UID: \"f3f306c2-bbbc-41bf-8628-5cb0f1998b11\") " Nov 01 09:53:09.504401 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.504377 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume" pod "f3f306c2-bbbc-41bf-8628-5cb0f1998b11" (UID: "f3f306c2-bbbc-41bf-8628-5cb0f1998b11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:53:09.504689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.504674 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:09.504823 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.504691 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dv5l\" (UniqueName: \"kubernetes.io/projected/919c821a-75eb-4d38-8722-5e7d19d5fd8d-kube-api-access-5dv5l\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:09.506800 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.506763 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82" pod "f3f306c2-bbbc-41bf-8628-5cb0f1998b11" (UID: "f3f306c2-bbbc-41bf-8628-5cb0f1998b11"). InnerVolumeSpecName "kube-api-access-4dg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:53:09.606551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:09.606500 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dg82\" (UniqueName: \"kubernetes.io/projected/f3f306c2-bbbc-41bf-8628-5cb0f1998b11-kube-api-access-4dg82\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:10.201222 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.201186 5710 generic.go:356] "Generic (PLEG): container finished" podID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerID="3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309" exitCode=0 Nov 01 09:53:10.201434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.201238 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" Nov 01 09:53:10.201434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.201273 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" event={"ID":"f3f306c2-bbbc-41bf-8628-5cb0f1998b11","Type":"ContainerDied","Data":"3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309"} Nov 01 09:53:10.201434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.201334 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5" event={"ID":"f3f306c2-bbbc-41bf-8628-5cb0f1998b11","Type":"ContainerDied","Data":"0cc0bea14e3b6bfc6aa394a62e3e8afdee5a24521bb367ebfee74913147c9935"} Nov 01 09:53:10.201434 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.201363 5710 scope.go:122] "RemoveContainer" containerID="3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309" Nov 01 09:53:10.203065 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.203041 5710 generic.go:356] "Generic (PLEG): container finished" podID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerID="79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772" exitCode=0 Nov 01 09:53:10.203197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.203091 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" event={"ID":"919c821a-75eb-4d38-8722-5e7d19d5fd8d","Type":"ContainerDied","Data":"79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772"} Nov 01 09:53:10.203197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.203101 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" Nov 01 09:53:10.203197 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.203130 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8" event={"ID":"919c821a-75eb-4d38-8722-5e7d19d5fd8d","Type":"ContainerDied","Data":"b4568c2c0df7970bcdb6e3a0c7704974081a1a4046095c54b2bd2edc16f36c92"} Nov 01 09:53:10.215373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.215344 5710 scope.go:122] "RemoveContainer" containerID="3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309" Nov 01 09:53:10.221039 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:10.220990 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309\": container with ID starting with 3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309 not found: ID does not exist" containerID="3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309" Nov 01 09:53:10.221226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.221035 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309"} err="failed to get container status \"3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309\": rpc error: code = NotFound desc = could not find container \"3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309\": container with ID starting with 3e3d52fc8f610bed551bc3184a85b8bfd9709b94a4768cd597c9aa6b2cc31309 not found: ID does not exist" Nov 01 09:53:10.221226 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.221067 5710 scope.go:122] "RemoveContainer" containerID="79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772" Nov 01 09:53:10.232921 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.232890 5710 scope.go:122] "RemoveContainer" containerID="79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772" Nov 01 09:53:10.233292 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:10.233268 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772\": container with ID starting with 79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772 not found: ID does not exist" containerID="79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772" Nov 01 09:53:10.233378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.233303 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772"} err="failed to get container status \"79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772\": rpc error: code = NotFound desc = could not find container \"79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772\": container with ID starting with 79f7b3f8ded652fea6f93eb099b98c5d568a830d3da157f785c22ac78c4b7772 not found: ID does not exist" Nov 01 09:53:10.363256 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.363221 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5"] Nov 01 09:53:10.409684 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.409594 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-035625-6c6cd8b655-ckld5"] Nov 01 09:53:10.484302 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.484257 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8"] Nov 01 09:53:10.491090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:10.491058 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-035625-55b4574d85-7ntv8"] Nov 01 09:53:11.058874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:11.058839 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" path="/var/lib/kubelet/pods/919c821a-75eb-4d38-8722-5e7d19d5fd8d/volumes" Nov 01 09:53:11.059296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:11.059281 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" path="/var/lib/kubelet/pods/f3f306c2-bbbc-41bf-8628-5cb0f1998b11/volumes" Nov 01 09:53:13.769798 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:13.769758 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:53:15.479110 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:15.479070 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:53:38.106396 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.106357 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc"] Nov 01 09:53:38.106882 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.106622 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" containerName="maas-api" containerID="cri-o://ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc" gracePeriod=30 Nov 01 09:53:38.354743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.353865 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh"] Nov 01 09:53:38.424347 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.424320 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:53:38.457116 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.457080 5710 generic.go:356] "Generic (PLEG): container finished" podID="242da1e1-32b9-401a-9f7d-f42b00b17609" containerID="ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc" exitCode=0 Nov 01 09:53:38.457294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.457135 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" Nov 01 09:53:38.457294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.457164 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" event={"ID":"242da1e1-32b9-401a-9f7d-f42b00b17609","Type":"ContainerDied","Data":"ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc"} Nov 01 09:53:38.457294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.457220 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc" event={"ID":"242da1e1-32b9-401a-9f7d-f42b00b17609","Type":"ContainerDied","Data":"3e8b16e0021cbcffb4f746438baa30077c5c0e15cba4f8244c3e8226626e3997"} Nov 01 09:53:38.457294 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.457242 5710 scope.go:122] "RemoveContainer" containerID="ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc" Nov 01 09:53:38.467584 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.467563 5710 scope.go:122] "RemoveContainer" containerID="ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc" Nov 01 09:53:38.467916 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:38.467890 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc\": container with ID starting with ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc not found: ID does not exist" containerID="ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc" Nov 01 09:53:38.468018 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.467934 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc"} err="failed to get container status \"ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc\": rpc error: code = NotFound desc = could not find container \"ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc\": container with ID starting with ab237ddc39632a89b2caf986d4febdc9740b7695e1380210aa6fcef704a071cc not found: ID does not exist" Nov 01 09:53:38.551519 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.551492 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt\" (UniqueName: \"kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt\") pod \"242da1e1-32b9-401a-9f7d-f42b00b17609\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " Nov 01 09:53:38.551767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.551551 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs\") pod \"242da1e1-32b9-401a-9f7d-f42b00b17609\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " Nov 01 09:53:38.551767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.551712 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\") pod \"242da1e1-32b9-401a-9f7d-f42b00b17609\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " Nov 01 09:53:38.551928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.551774 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca\") pod \"242da1e1-32b9-401a-9f7d-f42b00b17609\" (UID: \"242da1e1-32b9-401a-9f7d-f42b00b17609\") " Nov 01 09:53:38.552256 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.552180 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca" pod "242da1e1-32b9-401a-9f7d-f42b00b17609" (UID: "242da1e1-32b9-401a-9f7d-f42b00b17609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:53:38.553929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.553902 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls" pod "242da1e1-32b9-401a-9f7d-f42b00b17609" (UID: "242da1e1-32b9-401a-9f7d-f42b00b17609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:53:38.554052 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.553968 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt" pod "242da1e1-32b9-401a-9f7d-f42b00b17609" (UID: "242da1e1-32b9-401a-9f7d-f42b00b17609"). InnerVolumeSpecName "kube-api-access-kcxxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:53:38.554052 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.554006 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs" pod "242da1e1-32b9-401a-9f7d-f42b00b17609" (UID: "242da1e1-32b9-401a-9f7d-f42b00b17609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:53:38.654031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.653917 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:38.654031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.653959 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/242da1e1-32b9-401a-9f7d-f42b00b17609-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:38.654031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.653969 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxxt\" (UniqueName: \"kubernetes.io/projected/242da1e1-32b9-401a-9f7d-f42b00b17609-kube-api-access-kcxxt\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:38.654031 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.653979 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242da1e1-32b9-401a-9f7d-f42b00b17609-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:53:38.819749 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:38.797650 5710 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242da1e1_32b9_401a_9f7d_f42b00b17609.slice/crio-3e8b16e0021cbcffb4f746438baa30077c5c0e15cba4f8244c3e8226626e3997\": RecentStats: unable to find data in memory cache]" Nov 01 09:53:38.881066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.881029 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc"] Nov 01 09:53:38.884212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:38.884185 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5b96ac-6d6c5549d6-zlqrc"] Nov 01 09:53:39.058555 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.058442 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" path="/var/lib/kubelet/pods/242da1e1-32b9-401a-9f7d-f42b00b17609/volumes" Nov 01 09:53:39.989571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.989540 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c"] Nov 01 09:53:39.990306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990283 5710 cpu_manager.go:414] "containerMap: removing container" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:53:39.990306 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990304 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990334 5710 cpu_manager.go:414] "containerMap: removing container" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerName="payload-processing" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990340 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerName="payload-processing" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990361 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990366 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53917f1-1232-480e-b014-e81651753c9a" containerName="istio-proxy" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990379 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990383 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990393 5710 cpu_manager.go:414] "containerMap: removing container" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" containerName="maas-api" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990397 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" containerName="maas-api" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990406 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerName="payload-pre-processing" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990412 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerName="payload-pre-processing" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990496 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="919c821a-75eb-4d38-8722-5e7d19d5fd8d" containerName="payload-processing" Nov 01 09:53:39.990502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990511 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="242da1e1-32b9-401a-9f7d-f42b00b17609" containerName="maas-api" Nov 01 09:53:39.990920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990518 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f306c2-bbbc-41bf-8628-5cb0f1998b11" containerName="payload-pre-processing" Nov 01 09:53:39.990920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990524 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ca7164-be67-45b6-b5f2-b89535f0f70e" containerName="cleanup" Nov 01 09:53:39.990920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990531 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="182eb899-9d31-4514-b545-de771d99bc4b" containerName="istio-proxy" Nov 01 09:53:39.990920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990537 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b987afa-970c-4e07-93f2-a567ad2a7f2a" containerName="cleanup" Nov 01 09:53:39.990920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.990543 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="34951a7b-c05a-4e28-ad67-318a376b1616" containerName="cleanup" Nov 01 09:53:39.995567 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:39.995549 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.046486 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.046456 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0401bd23-serving-cert\"" Nov 01 09:53:40.060264 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.060230 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c"] Nov 01 09:53:40.076986 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.076951 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.077198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.077128 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.077198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.077182 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwml\" (UniqueName: \"kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.077297 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.077248 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.178378 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.178333 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.178575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.178393 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwml\" (UniqueName: \"kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.178575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.178434 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.178575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.178494 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.178702 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:40.178619 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-serving-cert: secret "maas-api-e2e-mt-0401bd23-serving-cert" not found Nov 01 09:53:40.178801 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:40.178711 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls podName:8f5dfc48-0342-4e80-90c1-5ddae3f4497e nodeName:}" failed. No retries permitted until 2026-11-01 09:53:40.678685652 +0000 UTC m=+4843.767022319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls") pod "maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" (UID: "8f5dfc48-0342-4e80-90c1-5ddae3f4497e") : secret "maas-api-e2e-mt-0401bd23-serving-cert" not found Nov 01 09:53:40.179200 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.179180 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.181180 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.181161 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.246163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.245916 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwml\" (UniqueName: \"kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.684150 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.684113 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.686868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.686841 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\") pod \"maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:40.910218 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:40.910179 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:41.350686 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:41.350652 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c"] Nov 01 09:53:41.353198 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:53:41.353169 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5dfc48_0342_4e80_90c1_5ddae3f4497e.slice/crio-13f1a814ac4133aa56eeae4bf27de42b82c63052a46df44d1c3e0c7cec399d39 WatchSource:0}: Error finding container 13f1a814ac4133aa56eeae4bf27de42b82c63052a46df44d1c3e0c7cec399d39: Status 404 returned error can't find the container with id 13f1a814ac4133aa56eeae4bf27de42b82c63052a46df44d1c3e0c7cec399d39 Nov 01 09:53:41.354498 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:41.354475 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 09:53:41.490491 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:41.490450 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" event={"ID":"8f5dfc48-0342-4e80-90c1-5ddae3f4497e","Type":"ContainerStarted","Data":"13f1a814ac4133aa56eeae4bf27de42b82c63052a46df44d1c3e0c7cec399d39"} Nov 01 09:53:42.499274 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:42.499230 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" event={"ID":"8f5dfc48-0342-4e80-90c1-5ddae3f4497e","Type":"ContainerStarted","Data":"4ec82d874726b555ca254e8a15a90b90316b124cc21117115ef78328234339ac"} Nov 01 09:53:42.499693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:42.499307 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:42.641276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:42.641214 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" podStartSLOduration=2.6024491960000002 podStartE2EDuration="3.641199597s" totalImagesPullingTime="1.038750401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:53:42.638668041 +0000 UTC m=+4845.727004715" watchObservedRunningTime="2026-11-01 09:53:42.641199597 +0000 UTC m=+4845.729536268" Nov 01 09:53:43.355190 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:43.355145 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerName="payload-processing" containerID="cri-o://f1960fa6850b8ac2432e47f28f28d94632f276fa8c79817a1bce70314369db36" gracePeriod=25 Nov 01 09:53:44.883002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:44.882959 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:53:45.477050 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:53:45.477005 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:53:48.512042 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:48.512011 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:53:54.882583 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:53:54.882541 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:54:03.635137 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.635100 5710 generic.go:356] "Generic (PLEG): container finished" podID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerID="f1960fa6850b8ac2432e47f28f28d94632f276fa8c79817a1bce70314369db36" exitCode=0 Nov 01 09:54:03.635560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.635171 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" event={"ID":"0db332be-99b7-4989-bd7e-c2c07a30ee53","Type":"ContainerDied","Data":"f1960fa6850b8ac2432e47f28f28d94632f276fa8c79817a1bce70314369db36"} Nov 01 09:54:03.635560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.635234 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" event={"ID":"0db332be-99b7-4989-bd7e-c2c07a30ee53","Type":"ContainerDied","Data":"063182b91a965b735e8de0242155ca271af95897234139b446efb1b731e6ea16"} Nov 01 09:54:03.635560 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.635254 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="063182b91a965b735e8de0242155ca271af95897234139b446efb1b731e6ea16" Nov 01 09:54:03.639090 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.639073 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:54:03.820302 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.820270 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq\" (UniqueName: \"kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq\") pod \"0db332be-99b7-4989-bd7e-c2c07a30ee53\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " Nov 01 09:54:03.820472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.820328 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume\") pod \"0db332be-99b7-4989-bd7e-c2c07a30ee53\" (UID: \"0db332be-99b7-4989-bd7e-c2c07a30ee53\") " Nov 01 09:54:03.820852 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.820824 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume" pod "0db332be-99b7-4989-bd7e-c2c07a30ee53" (UID: "0db332be-99b7-4989-bd7e-c2c07a30ee53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:54:03.822838 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.822811 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq" pod "0db332be-99b7-4989-bd7e-c2c07a30ee53" (UID: "0db332be-99b7-4989-bd7e-c2c07a30ee53"). InnerVolumeSpecName "kube-api-access-5mtsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:54:03.922322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.922229 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtsq\" (UniqueName: \"kubernetes.io/projected/0db332be-99b7-4989-bd7e-c2c07a30ee53-kube-api-access-5mtsq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:54:03.922322 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:03.922264 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db332be-99b7-4989-bd7e-c2c07a30ee53-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:54:04.641416 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:04.641381 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh" Nov 01 09:54:04.801659 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:04.801619 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh"] Nov 01 09:54:04.808394 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:04.808364 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5b96ac-754b6bf789-hzpdh"] Nov 01 09:54:05.068928 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:05.068832 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" path="/var/lib/kubelet/pods/0db332be-99b7-4989-bd7e-c2c07a30ee53/volumes" Nov 01 09:54:15.481811 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:54:15.481773 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:54:16.901252 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:16.901206 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:54:31.084953 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.084913 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq"] Nov 01 09:54:31.085701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.085681 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerName="payload-processing" Nov 01 09:54:31.085826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.085704 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerName="payload-processing" Nov 01 09:54:31.085826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.085821 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db332be-99b7-4989-bd7e-c2c07a30ee53" containerName="payload-processing" Nov 01 09:54:31.092524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.092496 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.119830 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.119791 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhwzn\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.119999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.119848 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.119999 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.119951 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.120086 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.120071 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.120131 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.120115 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.120169 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.120162 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.120239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.120225 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.120292 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.120278 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.120337 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.120324 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.144515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.144475 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a87a1d-openshift-default-dockercfg-jwv8b\"" Nov 01 09:54:31.149048 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.149019 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq"] Nov 01 09:54:31.221077 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221040 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221104 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhwzn\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221134 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221167 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221279 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221256 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221307 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221328 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221379 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221454 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221734 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221690 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221826 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221798 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.221889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221837 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.222010 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.221992 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.222247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.222114 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.223949 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.223921 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.224408 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.224391 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.281069 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.281035 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.281247 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.281134 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhwzn\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn\") pod \"e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.409087 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.409041 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:31.847845 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.847604 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq"] Nov 01 09:54:31.851232 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:54:31.851196 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ced5b6_3157_4b50_ade2_102964383121.slice/crio-8567812e6432b8e2e98ed04f2aa37f941f1f8c946802a5ebd0dcb35bba4720d4 WatchSource:0}: Error finding container 8567812e6432b8e2e98ed04f2aa37f941f1f8c946802a5ebd0dcb35bba4720d4: Status 404 returned error can't find the container with id 8567812e6432b8e2e98ed04f2aa37f941f1f8c946802a5ebd0dcb35bba4720d4 Nov 01 09:54:31.854082 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.854044 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:54:31.854163 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.854133 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:54:31.854201 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:31.854174 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:54:32.863310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:32.863257 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" event={"ID":"a4ced5b6-3157-4b50-ade2-102964383121","Type":"ContainerStarted","Data":"8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf"} Nov 01 09:54:32.863764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:32.863336 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" event={"ID":"a4ced5b6-3157-4b50-ade2-102964383121","Type":"ContainerStarted","Data":"8567812e6432b8e2e98ed04f2aa37f941f1f8c946802a5ebd0dcb35bba4720d4"} Nov 01 09:54:33.003743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:33.002704 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podStartSLOduration=2.002681209 podStartE2EDuration="2.002681209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:54:33.000417714 +0000 UTC m=+4896.088754414" watchObservedRunningTime="2026-11-01 09:54:33.002681209 +0000 UTC m=+4896.091017882" Nov 01 09:54:33.409363 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:33.409326 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:33.411402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:33.411370 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 01 09:54:33.411571 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:33.411430 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 01 09:54:34.409973 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:34.409936 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 01 09:54:34.410426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:34.409992 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 01 09:54:35.410260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:35.410217 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 01 09:54:35.410708 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:35.410273 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 01 09:54:36.409551 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:36.409510 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 01 09:54:36.409857 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:36.409567 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 01 09:54:37.409637 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:37.409597 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 01 09:54:37.410185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:37.409653 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 01 09:54:38.409485 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:38.409442 5710 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 01 09:54:38.409695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:38.409509 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 01 09:54:39.414711 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:39.414679 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:39.415191 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:39.414967 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:39.415914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:39.415894 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 09:54:45.481914 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:54:45.481861 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:54:46.313132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.313092 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv"] Nov 01 09:54:46.317635 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.317609 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.353814 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.353767 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.353904 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.353997 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354051 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.354035 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.354065 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.354155 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354194 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.354185 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxg5\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354296 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.354257 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.354407 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.354390 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.405923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.405885 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-46bf99-openshift-default-dockercfg-2rzjn\"" Nov 01 09:54:46.456037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.455990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456036 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456071 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456094 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxg5\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456126 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456253 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456340 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456310 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456367 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456435 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456621 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456522 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456713 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456853 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456757 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.456979 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.456871 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.457366 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.457341 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.458672 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.458651 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.458941 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.458924 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.484790 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.484755 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv"] Nov 01 09:54:46.520287 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.520255 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxg5\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.526944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.526911 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token\") pod \"e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:46.633291 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:46.633254 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:47.073185 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:47.073158 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv"] Nov 01 09:54:47.076756 ip-10-0-143-16 kubenswrapper[5710]: W1101 09:54:47.076694 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bc6462_8a3d_4b79_b3b1_cef9a1a93e05.slice/crio-04511e3cc1cc7176fd408a4b0b38daf163d48404dba80a7135a3e567195bfb5e WatchSource:0}: Error finding container 04511e3cc1cc7176fd408a4b0b38daf163d48404dba80a7135a3e567195bfb5e: Status 404 returned error can't find the container with id 04511e3cc1cc7176fd408a4b0b38daf163d48404dba80a7135a3e567195bfb5e Nov 01 09:54:47.079465 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:47.079433 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:54:47.079546 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:47.079498 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:54:47.079546 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:47.079526 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 09:54:47.978156 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:47.978112 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" event={"ID":"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05","Type":"ContainerStarted","Data":"eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742"} Nov 01 09:54:47.978657 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:47.978178 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" event={"ID":"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05","Type":"ContainerStarted","Data":"04511e3cc1cc7176fd408a4b0b38daf163d48404dba80a7135a3e567195bfb5e"} Nov 01 09:54:48.633694 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:48.633659 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:48.635754 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:48.635710 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 09:54:48.635942 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:48.635777 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 09:54:49.633570 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:49.633535 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 09:54:49.634037 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:49.633588 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 09:54:50.633557 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:50.633521 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 09:54:50.633775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:50.633580 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 09:54:51.634344 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:51.634310 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 09:54:51.634802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:51.634371 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 09:54:52.634030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:52.633991 5710 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 01 09:54:52.634225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:52.634044 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 01 09:54:53.857199 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:53.857155 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:53.857731 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:53.857641 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:53.858867 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:53.858844 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:54:54.021550 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:54:54.021494 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podStartSLOduration=8.021480067 podStartE2EDuration="8.021480067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 09:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 09:54:48.142777893 +0000 UTC m=+4911.231114563" watchObservedRunningTime="2026-11-01 09:54:54.021480067 +0000 UTC m=+4917.109816737" Nov 01 09:55:00.771145 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:00.771113 5710 scope.go:122] "RemoveContainer" containerID="f1960fa6850b8ac2432e47f28f28d94632f276fa8c79817a1bce70314369db36" Nov 01 09:55:15.482204 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:55:15.482162 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:55:28.023153 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:28.023098 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:55:29.846887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:29.846849 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44"] Nov 01 09:55:29.944025 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:29.943987 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv"] Nov 01 09:55:34.847796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:34.847650 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" podUID="93e1360d-f860-472d-a024-dae5df31ed42" containerName="payload-processing" containerID="cri-o://66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa" gracePeriod=25 Nov 01 09:55:34.944679 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:34.944614 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" podUID="e1131cda-c18a-4e17-a97a-771c88b65c9f" containerName="payload-pre-processing" containerID="cri-o://8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343" gracePeriod=25 Nov 01 09:55:35.123410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.123338 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 09:55:35.232132 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.232101 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 09:55:35.235693 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.235667 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume\") pod \"93e1360d-f860-472d-a024-dae5df31ed42\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " Nov 01 09:55:35.235868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.235781 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b\" (UniqueName: \"kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b\") pod \"93e1360d-f860-472d-a024-dae5df31ed42\" (UID: \"93e1360d-f860-472d-a024-dae5df31ed42\") " Nov 01 09:55:35.236310 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.236271 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume" pod "93e1360d-f860-472d-a024-dae5df31ed42" (UID: "93e1360d-f860-472d-a024-dae5df31ed42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:55:35.236474 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.236435 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e1360d-f860-472d-a024-dae5df31ed42-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:35.239036 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.239005 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b" pod "93e1360d-f860-472d-a024-dae5df31ed42" (UID: "93e1360d-f860-472d-a024-dae5df31ed42"). InnerVolumeSpecName "kube-api-access-nfs6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:55:35.337869 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.337840 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume\") pod \"e1131cda-c18a-4e17-a97a-771c88b65c9f\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " Nov 01 09:55:35.338068 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.338013 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5\" (UniqueName: \"kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5\") pod \"e1131cda-c18a-4e17-a97a-771c88b65c9f\" (UID: \"e1131cda-c18a-4e17-a97a-771c88b65c9f\") " Nov 01 09:55:35.338251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.338229 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume" pod "e1131cda-c18a-4e17-a97a-771c88b65c9f" (UID: "e1131cda-c18a-4e17-a97a-771c88b65c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:55:35.338537 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.338521 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1131cda-c18a-4e17-a97a-771c88b65c9f-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:35.338616 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.338539 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfs6b\" (UniqueName: \"kubernetes.io/projected/93e1360d-f860-472d-a024-dae5df31ed42-kube-api-access-nfs6b\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:35.340520 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.340491 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5" pod "e1131cda-c18a-4e17-a97a-771c88b65c9f" (UID: "e1131cda-c18a-4e17-a97a-771c88b65c9f"). InnerVolumeSpecName "kube-api-access-tr5j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:55:35.353954 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.353915 5710 generic.go:356] "Generic (PLEG): container finished" podID="e1131cda-c18a-4e17-a97a-771c88b65c9f" containerID="8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343" exitCode=0 Nov 01 09:55:35.354460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.354414 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" event={"ID":"e1131cda-c18a-4e17-a97a-771c88b65c9f","Type":"ContainerDied","Data":"8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343"} Nov 01 09:55:35.355966 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.355938 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" event={"ID":"e1131cda-c18a-4e17-a97a-771c88b65c9f","Type":"ContainerDied","Data":"d9297f3b54c21e48d2abf43f5904052106847e6d41ce9f724ea0424bdc7f18b6"} Nov 01 09:55:35.356166 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.354250 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv" Nov 01 09:55:35.357018 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.356052 5710 scope.go:122] "RemoveContainer" containerID="8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343" Nov 01 09:55:35.360125 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.360097 5710 generic.go:356] "Generic (PLEG): container finished" podID="93e1360d-f860-472d-a024-dae5df31ed42" containerID="66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa" exitCode=0 Nov 01 09:55:35.360441 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.360413 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" Nov 01 09:55:35.360798 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.360501 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" event={"ID":"93e1360d-f860-472d-a024-dae5df31ed42","Type":"ContainerDied","Data":"66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa"} Nov 01 09:55:35.360798 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.360562 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44" event={"ID":"93e1360d-f860-472d-a024-dae5df31ed42","Type":"ContainerDied","Data":"952cff860016c3de41efbc74ef1a124af6cfc994af64b88cfbf365637f66774c"} Nov 01 09:55:35.372524 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.372495 5710 scope.go:122] "RemoveContainer" containerID="8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343" Nov 01 09:55:35.373225 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:55:35.373198 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343\": container with ID starting with 8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343 not found: ID does not exist" containerID="8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343" Nov 01 09:55:35.373328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.373237 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343"} err="failed to get container status \"8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343\": rpc error: code = NotFound desc = could not find container \"8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343\": container with ID starting with 8d66f52e9579f83265ee74e3d096be7903a57c1c80a1fb24d9943add8be66343 not found: ID does not exist" Nov 01 09:55:35.373328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.373261 5710 scope.go:122] "RemoveContainer" containerID="66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa" Nov 01 09:55:35.382531 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.382508 5710 scope.go:122] "RemoveContainer" containerID="66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa" Nov 01 09:55:35.382830 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:55:35.382806 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa\": container with ID starting with 66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa not found: ID does not exist" containerID="66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa" Nov 01 09:55:35.382908 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.382841 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa"} err="failed to get container status \"66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa\": rpc error: code = NotFound desc = could not find container \"66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa\": container with ID starting with 66a6c9a300d5696bb8ea19b5185bb89ea03fdc192daf7e276e498d95c20defaa not found: ID does not exist" Nov 01 09:55:35.439373 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.439341 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5j5\" (UniqueName: \"kubernetes.io/projected/e1131cda-c18a-4e17-a97a-771c88b65c9f-kube-api-access-tr5j5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:35.513415 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.513380 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44"] Nov 01 09:55:35.517355 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.517323 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-343207-6bd4f48bdf-w5p44"] Nov 01 09:55:35.620874 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.620833 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv"] Nov 01 09:55:35.626073 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:35.625989 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-343207-7f8f9cb7-mwqhv"] Nov 01 09:55:37.058923 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:37.058891 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e1360d-f860-472d-a024-dae5df31ed42" path="/var/lib/kubelet/pods/93e1360d-f860-472d-a024-dae5df31ed42/volumes" Nov 01 09:55:37.059388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:37.059373 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1131cda-c18a-4e17-a97a-771c88b65c9f" path="/var/lib/kubelet/pods/e1131cda-c18a-4e17-a97a-771c88b65c9f/volumes" Nov 01 09:55:45.482331 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:55:45.482292 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:55:58.204564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.204524 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85"] Nov 01 09:55:58.205058 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.204780 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" podUID="af82bf01-0af4-4bdd-9f81-941504980db7" containerName="maas-api" containerID="cri-o://20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda" gracePeriod=30 Nov 01 09:55:58.524815 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.524786 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 09:55:58.539370 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.539334 5710 generic.go:356] "Generic (PLEG): container finished" podID="af82bf01-0af4-4bdd-9f81-941504980db7" containerID="20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda" exitCode=0 Nov 01 09:55:58.539564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.539386 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" Nov 01 09:55:58.539564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.539418 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" event={"ID":"af82bf01-0af4-4bdd-9f81-941504980db7","Type":"ContainerDied","Data":"20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda"} Nov 01 09:55:58.539564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.539482 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85" event={"ID":"af82bf01-0af4-4bdd-9f81-941504980db7","Type":"ContainerDied","Data":"4955c6731e18984a57e9d948126ca4e7b99275d1dc21b458008f580e5f6c448c"} Nov 01 09:55:58.539564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.539514 5710 scope.go:122] "RemoveContainer" containerID="20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda" Nov 01 09:55:58.549911 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.549889 5710 scope.go:122] "RemoveContainer" containerID="20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda" Nov 01 09:55:58.550248 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:55:58.550227 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda\": container with ID starting with 20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda not found: ID does not exist" containerID="20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda" Nov 01 09:55:58.550323 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.550262 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda"} err="failed to get container status \"20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda\": rpc error: code = NotFound desc = could not find container \"20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda\": container with ID starting with 20e7ee1373309ed80a4cdf619c4293bac44e95061c974a54f632b5eb6fc2dbda not found: ID does not exist" Nov 01 09:55:58.570277 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.570237 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca\") pod \"af82bf01-0af4-4bdd-9f81-941504980db7\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " Nov 01 09:55:58.570471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.570358 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls\") pod \"af82bf01-0af4-4bdd-9f81-941504980db7\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " Nov 01 09:55:58.570471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.570465 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j\" (UniqueName: \"kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j\") pod \"af82bf01-0af4-4bdd-9f81-941504980db7\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " Nov 01 09:55:58.570589 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.570500 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs\") pod \"af82bf01-0af4-4bdd-9f81-941504980db7\" (UID: \"af82bf01-0af4-4bdd-9f81-941504980db7\") " Nov 01 09:55:58.570642 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.570621 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca" pod "af82bf01-0af4-4bdd-9f81-941504980db7" (UID: "af82bf01-0af4-4bdd-9f81-941504980db7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:55:58.571111 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.571055 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af82bf01-0af4-4bdd-9f81-941504980db7-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:58.573078 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.573048 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls" pod "af82bf01-0af4-4bdd-9f81-941504980db7" (UID: "af82bf01-0af4-4bdd-9f81-941504980db7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:55:58.573253 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.573164 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j" pod "af82bf01-0af4-4bdd-9f81-941504980db7" (UID: "af82bf01-0af4-4bdd-9f81-941504980db7"). InnerVolumeSpecName "kube-api-access-bfm4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:55:58.623349 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.606869 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs" pod "af82bf01-0af4-4bdd-9f81-941504980db7" (UID: "af82bf01-0af4-4bdd-9f81-941504980db7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:55:58.672775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.672708 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:58.672775 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.672773 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfm4j\" (UniqueName: \"kubernetes.io/projected/af82bf01-0af4-4bdd-9f81-941504980db7-kube-api-access-bfm4j\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:58.672996 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.672785 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af82bf01-0af4-4bdd-9f81-941504980db7-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:55:58.963490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.963445 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85"] Nov 01 09:55:58.967311 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:58.967281 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-343207-ddffdb6d8-2dg85"] Nov 01 09:55:59.058439 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:55:59.058399 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af82bf01-0af4-4bdd-9f81-941504980db7" path="/var/lib/kubelet/pods/af82bf01-0af4-4bdd-9f81-941504980db7/volumes" Nov 01 09:56:14.035517 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:14.035480 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:56:15.479712 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:56:15.479669 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:56:34.217765 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:34.217706 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b"] Nov 01 09:56:39.221917 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.221860 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" podUID="b5512ce5-e232-4b9b-b8f5-19b0300f2f07" containerName="payload-pre-processing" containerID="cri-o://c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8" gracePeriod=25 Nov 01 09:56:39.530901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.530868 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:56:39.582903 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.582871 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume\") pod \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " Nov 01 09:56:39.583182 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.583162 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz\" (UniqueName: \"kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz\") pod \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\" (UID: \"b5512ce5-e232-4b9b-b8f5-19b0300f2f07\") " Nov 01 09:56:39.583351 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.583321 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume" pod "b5512ce5-e232-4b9b-b8f5-19b0300f2f07" (UID: "b5512ce5-e232-4b9b-b8f5-19b0300f2f07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:56:39.583764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.583699 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:56:39.585926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.585893 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz" pod "b5512ce5-e232-4b9b-b8f5-19b0300f2f07" (UID: "b5512ce5-e232-4b9b-b8f5-19b0300f2f07"). InnerVolumeSpecName "kube-api-access-6bnpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:56:39.688426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.688361 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnpz\" (UniqueName: \"kubernetes.io/projected/b5512ce5-e232-4b9b-b8f5-19b0300f2f07-kube-api-access-6bnpz\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:56:39.824195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.824083 5710 generic.go:356] "Generic (PLEG): container finished" podID="b5512ce5-e232-4b9b-b8f5-19b0300f2f07" containerID="c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8" exitCode=0 Nov 01 09:56:39.824195 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.824164 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" Nov 01 09:56:39.824469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.824160 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" event={"ID":"b5512ce5-e232-4b9b-b8f5-19b0300f2f07","Type":"ContainerDied","Data":"c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8"} Nov 01 09:56:39.824469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.824331 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b" event={"ID":"b5512ce5-e232-4b9b-b8f5-19b0300f2f07","Type":"ContainerDied","Data":"09c1b7ae53f22052f56babf9feca8e4bc6c2438983d6020048c3c15818183578"} Nov 01 09:56:39.824469 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.824371 5710 scope.go:122] "RemoveContainer" containerID="c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8" Nov 01 09:56:39.838648 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.838622 5710 scope.go:122] "RemoveContainer" containerID="c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8" Nov 01 09:56:39.839061 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:56:39.839031 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8\": container with ID starting with c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8 not found: ID does not exist" containerID="c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8" Nov 01 09:56:39.839172 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.839073 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8"} err="failed to get container status \"c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8\": rpc error: code = NotFound desc = could not find container \"c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8\": container with ID starting with c9128412bce2485d4ae4cdb354b8c4df6292351e2efb519768e8fcfcd94b50d8 not found: ID does not exist" Nov 01 09:56:39.980369 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.980331 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b"] Nov 01 09:56:39.984509 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:39.984475 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0401bd23-6db5c4bdbc-m4l8b"] Nov 01 09:56:41.059105 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:41.059068 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5512ce5-e232-4b9b-b8f5-19b0300f2f07" path="/var/lib/kubelet/pods/b5512ce5-e232-4b9b-b8f5-19b0300f2f07/volumes" Nov 01 09:56:43.708446 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:43.708406 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c"] Nov 01 09:56:43.708929 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:43.708637 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" podUID="8f5dfc48-0342-4e80-90c1-5ddae3f4497e" containerName="maas-api" containerID="cri-o://4ec82d874726b555ca254e8a15a90b90316b124cc21117115ef78328234339ac" gracePeriod=30 Nov 01 09:56:43.854502 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:43.854465 5710 generic.go:356] "Generic (PLEG): container finished" podID="8f5dfc48-0342-4e80-90c1-5ddae3f4497e" containerID="4ec82d874726b555ca254e8a15a90b90316b124cc21117115ef78328234339ac" exitCode=0 Nov 01 09:56:43.854681 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:43.854638 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" event={"ID":"8f5dfc48-0342-4e80-90c1-5ddae3f4497e","Type":"ContainerDied","Data":"4ec82d874726b555ca254e8a15a90b90316b124cc21117115ef78328234339ac"} Nov 01 09:56:44.022012 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.021980 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:56:44.152326 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.152258 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca\") pod \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " Nov 01 09:56:44.152589 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.152558 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml\" (UniqueName: \"kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml\") pod \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " Nov 01 09:56:44.152701 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.152670 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs\") pod \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " Nov 01 09:56:44.152820 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.152735 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\") pod \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\" (UID: \"8f5dfc48-0342-4e80-90c1-5ddae3f4497e\") " Nov 01 09:56:44.156631 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.156575 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls" pod "8f5dfc48-0342-4e80-90c1-5ddae3f4497e" (UID: "8f5dfc48-0342-4e80-90c1-5ddae3f4497e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:56:44.156909 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.156880 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca" pod "8f5dfc48-0342-4e80-90c1-5ddae3f4497e" (UID: "8f5dfc48-0342-4e80-90c1-5ddae3f4497e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:56:44.158887 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.158848 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml" pod "8f5dfc48-0342-4e80-90c1-5ddae3f4497e" (UID: "8f5dfc48-0342-4e80-90c1-5ddae3f4497e"). InnerVolumeSpecName "kube-api-access-vhwml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:56:44.160127 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.160077 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs" pod "8f5dfc48-0342-4e80-90c1-5ddae3f4497e" (UID: "8f5dfc48-0342-4e80-90c1-5ddae3f4497e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:56:44.262865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.262582 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:56:44.262865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.262634 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwml\" (UniqueName: \"kubernetes.io/projected/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-kube-api-access-vhwml\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:56:44.262865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.262652 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:56:44.262865 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.262669 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f5dfc48-0342-4e80-90c1-5ddae3f4497e-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:56:44.869208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.869169 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" Nov 01 09:56:44.869697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.869171 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c" event={"ID":"8f5dfc48-0342-4e80-90c1-5ddae3f4497e","Type":"ContainerDied","Data":"13f1a814ac4133aa56eeae4bf27de42b82c63052a46df44d1c3e0c7cec399d39"} Nov 01 09:56:44.869697 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:44.869333 5710 scope.go:122] "RemoveContainer" containerID="4ec82d874726b555ca254e8a15a90b90316b124cc21117115ef78328234339ac" Nov 01 09:56:45.023976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:45.023934 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c"] Nov 01 09:56:45.070278 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:45.070239 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0401bd23-7ffdb68856-tvb4c"] Nov 01 09:56:45.478243 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:56:45.478206 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:56:47.060699 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:47.060667 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5dfc48-0342-4e80-90c1-5ddae3f4497e" path="/var/lib/kubelet/pods/8f5dfc48-0342-4e80-90c1-5ddae3f4497e/volumes" Nov 01 09:56:51.126971 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:56:51.126928 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:57:00.102257 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.102221 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf"] Nov 01 09:57:00.102769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.102425 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" podUID="0b61e9c7-f611-4cfd-a401-66225c0d1d61" containerName="maas-api" containerID="cri-o://bff47b3d7c138f3442d5157462326bea57c18087e1eb6ebacf4eb16f5c52ca86" gracePeriod=30 Nov 01 09:57:00.412113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.412084 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:57:00.510353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.509039 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca\") pod \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " Nov 01 09:57:00.510353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.509130 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs\") pod \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " Nov 01 09:57:00.510353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.509173 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\") pod \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " Nov 01 09:57:00.510353 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.509232 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h\" (UniqueName: \"kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h\") pod \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\" (UID: \"0b61e9c7-f611-4cfd-a401-66225c0d1d61\") " Nov 01 09:57:00.512564 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.512229 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca" pod "0b61e9c7-f611-4cfd-a401-66225c0d1d61" (UID: "0b61e9c7-f611-4cfd-a401-66225c0d1d61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:57:00.525836 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.525792 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs" pod "0b61e9c7-f611-4cfd-a401-66225c0d1d61" (UID: "0b61e9c7-f611-4cfd-a401-66225c0d1d61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:57:00.539242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.536941 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h" pod "0b61e9c7-f611-4cfd-a401-66225c0d1d61" (UID: "0b61e9c7-f611-4cfd-a401-66225c0d1d61"). InnerVolumeSpecName "kube-api-access-nbt9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:00.544764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.543070 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls" pod "0b61e9c7-f611-4cfd-a401-66225c0d1d61" (UID: "0b61e9c7-f611-4cfd-a401-66225c0d1d61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:57:00.611443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.611402 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b61e9c7-f611-4cfd-a401-66225c0d1d61-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:00.611443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.611439 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:00.611443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.611448 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b61e9c7-f611-4cfd-a401-66225c0d1d61-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:00.611695 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.611457 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbt9h\" (UniqueName: \"kubernetes.io/projected/0b61e9c7-f611-4cfd-a401-66225c0d1d61-kube-api-access-nbt9h\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:00.822151 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.822065 5710 scope.go:122] "RemoveContainer" containerID="bff47b3d7c138f3442d5157462326bea57c18087e1eb6ebacf4eb16f5c52ca86" Nov 01 09:57:00.969135 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.969091 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" event={"ID":"0b61e9c7-f611-4cfd-a401-66225c0d1d61","Type":"ContainerDied","Data":"bff47b3d7c138f3442d5157462326bea57c18087e1eb6ebacf4eb16f5c52ca86"} Nov 01 09:57:00.969328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.969168 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" Nov 01 09:57:00.969328 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:00.969190 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf" event={"ID":"0b61e9c7-f611-4cfd-a401-66225c0d1d61","Type":"ContainerDied","Data":"934776c63e45704a2ec9e7edc7cdda789a36be50817add514fdb7f3c764df55d"} Nov 01 09:57:01.186104 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:01.186063 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf"] Nov 01 09:57:01.193628 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:01.193595 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-58de5a-6d859cf79b-5x5lf"] Nov 01 09:57:03.059113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:03.059081 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b61e9c7-f611-4cfd-a401-66225c0d1d61" path="/var/lib/kubelet/pods/0b61e9c7-f611-4cfd-a401-66225c0d1d61/volumes" Nov 01 09:57:09.876831 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:09.876792 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z"] Nov 01 09:57:14.047161 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:14.047121 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:57:14.877803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:14.877707 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" podUID="5d6be58b-baed-445c-a199-aee6ae24314a" containerName="payload-processing" containerID="cri-o://ac037825b4ee9ad7f0bbfeff435a38738f0c17ee6733afea54d44ab0c0e15925" gracePeriod=25 Nov 01 09:57:15.074463 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.074428 5710 generic.go:356] "Generic (PLEG): container finished" podID="5d6be58b-baed-445c-a199-aee6ae24314a" containerID="ac037825b4ee9ad7f0bbfeff435a38738f0c17ee6733afea54d44ab0c0e15925" exitCode=0 Nov 01 09:57:15.074907 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.074466 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" event={"ID":"5d6be58b-baed-445c-a199-aee6ae24314a","Type":"ContainerDied","Data":"ac037825b4ee9ad7f0bbfeff435a38738f0c17ee6733afea54d44ab0c0e15925"} Nov 01 09:57:15.161500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.161473 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:57:15.222495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.222465 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume\") pod \"5d6be58b-baed-445c-a199-aee6ae24314a\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " Nov 01 09:57:15.222683 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.222623 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66\" (UniqueName: \"kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66\") pod \"5d6be58b-baed-445c-a199-aee6ae24314a\" (UID: \"5d6be58b-baed-445c-a199-aee6ae24314a\") " Nov 01 09:57:15.222885 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.222865 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume" pod "5d6be58b-baed-445c-a199-aee6ae24314a" (UID: "5d6be58b-baed-445c-a199-aee6ae24314a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:57:15.223047 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.223027 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6be58b-baed-445c-a199-aee6ae24314a-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:15.225043 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.225023 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66" pod "5d6be58b-baed-445c-a199-aee6ae24314a" (UID: "5d6be58b-baed-445c-a199-aee6ae24314a"). InnerVolumeSpecName "kube-api-access-87j66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:15.324203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:15.324164 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87j66\" (UniqueName: \"kubernetes.io/projected/5d6be58b-baed-445c-a199-aee6ae24314a-kube-api-access-87j66\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:15.479125 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:57:15.479028 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:57:16.082540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:16.082496 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" event={"ID":"5d6be58b-baed-445c-a199-aee6ae24314a","Type":"ContainerDied","Data":"0c56261b04e56f5256f13b9ee036f72502359cfc7b178b5f8fac7c06bc0e2bce"} Nov 01 09:57:16.082540 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:16.082537 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z" Nov 01 09:57:16.083183 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:16.082575 5710 scope.go:122] "RemoveContainer" containerID="ac037825b4ee9ad7f0bbfeff435a38738f0c17ee6733afea54d44ab0c0e15925" Nov 01 09:57:16.236648 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:16.236616 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z"] Nov 01 09:57:16.283071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:16.283031 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a83714-864b496bdc-ws84z"] Nov 01 09:57:17.060225 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:17.060195 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6be58b-baed-445c-a199-aee6ae24314a" path="/var/lib/kubelet/pods/5d6be58b-baed-445c-a199-aee6ae24314a/volumes" Nov 01 09:57:32.676743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:32.674132 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9"] Nov 01 09:57:32.676743 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:32.674437 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" containerID="cri-o://340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d" gracePeriod=30 Nov 01 09:57:32.703792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:32.700855 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd"] Nov 01 09:57:32.703792 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:32.701110 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" containerID="cri-o://80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71" gracePeriod=30 Nov 01 09:57:37.991239 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:37.991151 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:57:38.016066 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.016037 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:57:38.019293 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019258 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.019477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019299 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019338 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019391 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019423 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019477 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019460 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019496 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019584 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019597 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019631 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019679 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019747 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.019802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019779 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019817 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019880 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019916 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019960 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w\") pod \"89009dff-0b02-4d79-96a0-ac92c3bbe881\" (UID: \"89009dff-0b02-4d79-96a0-ac92c3bbe881\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019998 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.019998 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.020003 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.020120 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data\") pod \"c5e69810-df19-492c-90bd-9d56ddc2d160\" (UID: \"c5e69810-df19-492c-90bd-9d56ddc2d160\") " Nov 01 09:57:38.020409 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.020313 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:57:38.020943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.020573 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.020943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.020780 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.020943 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.020857 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.021113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021004 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.021113 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021104 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.021244 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021229 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021249 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021263 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021278 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021292 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89009dff-0b02-4d79-96a0-ac92c3bbe881-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021309 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021306 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e69810-df19-492c-90bd-9d56ddc2d160-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021321 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021335 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.021587 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021351 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.022243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.021582 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.022333 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.022312 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:57:38.022843 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.022814 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.022968 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.022894 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:57:38.023385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.023355 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:57:38.023495 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.023447 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:38.023816 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.023799 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "kube-api-access-p748w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:38.024023 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.024006 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token" pod "89009dff-0b02-4d79-96a0-ac92c3bbe881" (UID: "89009dff-0b02-4d79-96a0-ac92c3bbe881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:38.024612 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.024594 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr" pod "c5e69810-df19-492c-90bd-9d56ddc2d160" (UID: "c5e69810-df19-492c-90bd-9d56ddc2d160"). InnerVolumeSpecName "kube-api-access-895pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:38.122585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122513 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122549 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122566 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122585 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122581 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122596 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p748w\" (UniqueName: \"kubernetes.io/projected/89009dff-0b02-4d79-96a0-ac92c3bbe881-kube-api-access-p748w\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122608 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e69810-df19-492c-90bd-9d56ddc2d160-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122617 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895pr\" (UniqueName: \"kubernetes.io/projected/c5e69810-df19-492c-90bd-9d56ddc2d160-kube-api-access-895pr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122625 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.122918 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.122636 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89009dff-0b02-4d79-96a0-ac92c3bbe881-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:38.220796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.220748 5710 generic.go:356] "Generic (PLEG): container finished" podID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerID="80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71" exitCode=0 Nov 01 09:57:38.220796 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.220794 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" Nov 01 09:57:38.221053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.220822 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" event={"ID":"89009dff-0b02-4d79-96a0-ac92c3bbe881","Type":"ContainerDied","Data":"80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71"} Nov 01 09:57:38.221053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.220881 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd" event={"ID":"89009dff-0b02-4d79-96a0-ac92c3bbe881","Type":"ContainerDied","Data":"75500beb8454c1e176b02845d7e6716b15b4f433b7f211f6cbfd483c6977d4a3"} Nov 01 09:57:38.221053 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.220908 5710 scope.go:122] "RemoveContainer" containerID="80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71" Nov 01 09:57:38.222395 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.222374 5710 generic.go:356] "Generic (PLEG): container finished" podID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerID="340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d" exitCode=0 Nov 01 09:57:38.222572 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.222425 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" Nov 01 09:57:38.222572 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.222460 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" event={"ID":"c5e69810-df19-492c-90bd-9d56ddc2d160","Type":"ContainerDied","Data":"340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d"} Nov 01 09:57:38.222572 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.222515 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9" event={"ID":"c5e69810-df19-492c-90bd-9d56ddc2d160","Type":"ContainerDied","Data":"f65a754d2306930f39d4278c09493402c9240e34b5dbe11d6ef15bd486640d63"} Nov 01 09:57:38.233160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.233137 5710 scope.go:122] "RemoveContainer" containerID="80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71" Nov 01 09:57:38.233513 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:57:38.233494 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71\": container with ID starting with 80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71 not found: ID does not exist" containerID="80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71" Nov 01 09:57:38.233593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.233522 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71"} err="failed to get container status \"80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71\": rpc error: code = NotFound desc = could not find container \"80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71\": container with ID starting with 80e76ef611e0e1d7a4f23c97ec4bdfdf327cee5ce10ab406034aee0f6afcff71 not found: ID does not exist" Nov 01 09:57:38.233593 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.233539 5710 scope.go:122] "RemoveContainer" containerID="340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d" Nov 01 09:57:38.244282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.244255 5710 scope.go:122] "RemoveContainer" containerID="340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d" Nov 01 09:57:38.244616 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:57:38.244595 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d\": container with ID starting with 340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d not found: ID does not exist" containerID="340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d" Nov 01 09:57:38.244667 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.244627 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d"} err="failed to get container status \"340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d\": rpc error: code = NotFound desc = could not find container \"340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d\": container with ID starting with 340a4920016087754d7e2c031ff2f63d7ea704c3572d96cfcb0617d272e9572d not found: ID does not exist" Nov 01 09:57:38.373176 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.373142 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd"] Nov 01 09:57:38.380940 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.380904 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vbndd"] Nov 01 09:57:38.484562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.484526 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9"] Nov 01 09:57:38.489260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:38.489229 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rjfq9"] Nov 01 09:57:39.059019 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:39.058942 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" path="/var/lib/kubelet/pods/89009dff-0b02-4d79-96a0-ac92c3bbe881/volumes" Nov 01 09:57:39.059460 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:39.059444 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" path="/var/lib/kubelet/pods/c5e69810-df19-492c-90bd-9d56ddc2d160/volumes" Nov 01 09:57:41.114730 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:41.114680 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:57:45.479561 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:57:45.479522 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:57:50.331097 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.331056 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg"] Nov 01 09:57:50.331602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.331296 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" podUID="28237a9a-dc8b-4c73-957f-41e47e14689c" containerName="maas-api" containerID="cri-o://bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5" gracePeriod=30 Nov 01 09:57:50.637794 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.637767 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:57:50.789248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.789202 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\") pod \"28237a9a-dc8b-4c73-957f-41e47e14689c\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " Nov 01 09:57:50.789445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.789276 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca\") pod \"28237a9a-dc8b-4c73-957f-41e47e14689c\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " Nov 01 09:57:50.789445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.789301 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs\") pod \"28237a9a-dc8b-4c73-957f-41e47e14689c\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " Nov 01 09:57:50.789445 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.789347 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q\" (UniqueName: \"kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q\") pod \"28237a9a-dc8b-4c73-957f-41e47e14689c\" (UID: \"28237a9a-dc8b-4c73-957f-41e47e14689c\") " Nov 01 09:57:50.789797 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.789760 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca" pod "28237a9a-dc8b-4c73-957f-41e47e14689c" (UID: "28237a9a-dc8b-4c73-957f-41e47e14689c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:57:50.791803 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.791766 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs" pod "28237a9a-dc8b-4c73-957f-41e47e14689c" (UID: "28237a9a-dc8b-4c73-957f-41e47e14689c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:57:50.791927 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.791850 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q" pod "28237a9a-dc8b-4c73-957f-41e47e14689c" (UID: "28237a9a-dc8b-4c73-957f-41e47e14689c"). InnerVolumeSpecName "kube-api-access-vbb5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:57:50.792095 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.792072 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls" pod "28237a9a-dc8b-4c73-957f-41e47e14689c" (UID: "28237a9a-dc8b-4c73-957f-41e47e14689c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:57:50.891255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.891154 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:50.891255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.891190 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28237a9a-dc8b-4c73-957f-41e47e14689c-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:50.891255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.891200 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28237a9a-dc8b-4c73-957f-41e47e14689c-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:50.891255 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:50.891209 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbb5q\" (UniqueName: \"kubernetes.io/projected/28237a9a-dc8b-4c73-957f-41e47e14689c-kube-api-access-vbb5q\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:57:51.330742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.330610 5710 generic.go:356] "Generic (PLEG): container finished" podID="28237a9a-dc8b-4c73-957f-41e47e14689c" containerID="bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5" exitCode=0 Nov 01 09:57:51.330742 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.330694 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" Nov 01 09:57:51.330975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.330740 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" event={"ID":"28237a9a-dc8b-4c73-957f-41e47e14689c","Type":"ContainerDied","Data":"bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5"} Nov 01 09:57:51.330975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.330813 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg" event={"ID":"28237a9a-dc8b-4c73-957f-41e47e14689c","Type":"ContainerDied","Data":"bc6f8429858999ef00e9c3f6001663a592efb1ddc4ca413be96c8def7f27f737"} Nov 01 09:57:51.330975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.330845 5710 scope.go:122] "RemoveContainer" containerID="bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5" Nov 01 09:57:51.340767 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.340709 5710 scope.go:122] "RemoveContainer" containerID="bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5" Nov 01 09:57:51.341111 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:57:51.341069 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5\": container with ID starting with bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5 not found: ID does not exist" containerID="bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5" Nov 01 09:57:51.341168 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.341104 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5"} err="failed to get container status \"bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5\": rpc error: code = NotFound desc = could not find container \"bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5\": container with ID starting with bbf1e818838dc8168b14a9a7a31af62ab5757ae661a9c3335c7acc080c8c95b5 not found: ID does not exist" Nov 01 09:57:51.485492 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.485458 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg"] Nov 01 09:57:51.511442 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:51.511402 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a83714-7c59d87bf5-qzgsg"] Nov 01 09:57:53.058950 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:57:53.058916 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28237a9a-dc8b-4c73-957f-41e47e14689c" path="/var/lib/kubelet/pods/28237a9a-dc8b-4c73-957f-41e47e14689c/volumes" Nov 01 09:58:01.372989 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:01.372961 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:58:01.383577 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:01.383551 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 09:58:15.478116 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:58:15.478069 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:58:16.153275 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:16.153236 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:58:18.253515 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:18.253478 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk"] Nov 01 09:58:18.369134 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:18.369102 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm"] Nov 01 09:58:23.254243 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:23.254176 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerName="payload-processing" containerID="cri-o://ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349" gracePeriod=25 Nov 01 09:58:23.370248 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:23.370211 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerName="payload-pre-processing" containerID="cri-o://b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2" gracePeriod=25 Nov 01 09:58:27.684021 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:27.683977 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp"] Nov 01 09:58:29.724245 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:29.724206 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:58:29.724763 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:29.724209 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:58:32.684807 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:32.684766 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerName="payload-processing" containerID="cri-o://73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80" gracePeriod=25 Nov 01 09:58:35.778933 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:35.778818 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:58:39.724038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:39.723996 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:58:39.724038 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:39.724022 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:58:43.547957 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.547924 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 09:58:43.560802 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.560774 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl\" (UniqueName: \"kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl\") pod \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " Nov 01 09:58:43.560978 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.560956 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume\") pod \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\" (UID: \"e1a8542e-ffb0-4dc5-b770-85d3205df3de\") " Nov 01 09:58:43.561417 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.561382 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume" pod "e1a8542e-ffb0-4dc5-b770-85d3205df3de" (UID: "e1a8542e-ffb0-4dc5-b770-85d3205df3de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:58:43.563119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.563095 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl" pod "e1a8542e-ffb0-4dc5-b770-85d3205df3de" (UID: "e1a8542e-ffb0-4dc5-b770-85d3205df3de"). InnerVolumeSpecName "kube-api-access-994dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:58:43.650212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.650186 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 09:58:43.662656 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.662616 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume\") pod \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " Nov 01 09:58:43.662901 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.662834 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf\" (UniqueName: \"kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf\") pod \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\" (UID: \"0d357506-a8cc-4c6d-bdd5-1bc71f53045c\") " Nov 01 09:58:43.663198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.663171 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume" pod "0d357506-a8cc-4c6d-bdd5-1bc71f53045c" (UID: "0d357506-a8cc-4c6d-bdd5-1bc71f53045c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:58:43.663404 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.663381 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a8542e-ffb0-4dc5-b770-85d3205df3de-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:43.663521 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.663409 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-994dl\" (UniqueName: \"kubernetes.io/projected/e1a8542e-ffb0-4dc5-b770-85d3205df3de-kube-api-access-994dl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:43.665420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.665395 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf" pod "0d357506-a8cc-4c6d-bdd5-1bc71f53045c" (UID: "0d357506-a8cc-4c6d-bdd5-1bc71f53045c"). InnerVolumeSpecName "kube-api-access-j26bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:58:43.668282 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.668250 5710 generic.go:356] "Generic (PLEG): container finished" podID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerID="ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349" exitCode=0 Nov 01 09:58:43.668402 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.668311 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" Nov 01 09:58:43.668510 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.668489 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" event={"ID":"e1a8542e-ffb0-4dc5-b770-85d3205df3de","Type":"ContainerDied","Data":"ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349"} Nov 01 09:58:43.668573 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.668541 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk" event={"ID":"e1a8542e-ffb0-4dc5-b770-85d3205df3de","Type":"ContainerDied","Data":"18284399245ba13c7e758e7d693791c0ab696dac9dab26447a05c33edd2875be"} Nov 01 09:58:43.668636 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.668572 5710 scope.go:122] "RemoveContainer" containerID="ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349" Nov 01 09:58:43.671298 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.671269 5710 generic.go:356] "Generic (PLEG): container finished" podID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerID="b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2" exitCode=0 Nov 01 09:58:43.671436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.671329 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" Nov 01 09:58:43.671436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.671338 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" event={"ID":"0d357506-a8cc-4c6d-bdd5-1bc71f53045c","Type":"ContainerDied","Data":"b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2"} Nov 01 09:58:43.671436 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.671382 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm" event={"ID":"0d357506-a8cc-4c6d-bdd5-1bc71f53045c","Type":"ContainerDied","Data":"ce57dc86c5d903d25f51e02ac7b887ce8b0271f18f129f0e8844ea7c80cac62d"} Nov 01 09:58:43.684071 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.684041 5710 scope.go:122] "RemoveContainer" containerID="ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349" Nov 01 09:58:43.685081 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:58:43.685055 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349\": container with ID starting with ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349 not found: ID does not exist" containerID="ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349" Nov 01 09:58:43.685210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.685089 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349"} err="failed to get container status \"ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349\": rpc error: code = NotFound desc = could not find container \"ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349\": container with ID starting with ca11a622ef9f8f7424af1dcdfee0923a90ced4c43d1f323f5d76cc9999eaa349 not found: ID does not exist" Nov 01 09:58:43.685210 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.685107 5710 scope.go:122] "RemoveContainer" containerID="b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2" Nov 01 09:58:43.696002 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.695973 5710 scope.go:122] "RemoveContainer" containerID="b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2" Nov 01 09:58:43.696379 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:58:43.696357 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2\": container with ID starting with b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2 not found: ID does not exist" containerID="b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2" Nov 01 09:58:43.696475 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.696391 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2"} err="failed to get container status \"b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2\": rpc error: code = NotFound desc = could not find container \"b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2\": container with ID starting with b2704d3aa9d87899eda46dd41996674ce3c06c512666f0b15b189f44e49ccba2 not found: ID does not exist" Nov 01 09:58:43.764273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.764231 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26bf\" (UniqueName: \"kubernetes.io/projected/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-kube-api-access-j26bf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:43.764273 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.764268 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d357506-a8cc-4c6d-bdd5-1bc71f53045c-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:43.874916 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.874877 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk"] Nov 01 09:58:43.880392 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.880353 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-751a73-7bbfdfdd8c-6tcxk"] Nov 01 09:58:43.981562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.981524 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm"] Nov 01 09:58:43.988385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:43.988354 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-751a73-96584f8c5-wrmmm"] Nov 01 09:58:45.058665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:45.058632 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" path="/var/lib/kubelet/pods/0d357506-a8cc-4c6d-bdd5-1bc71f53045c/volumes" Nov 01 09:58:45.059099 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:45.059076 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" path="/var/lib/kubelet/pods/e1a8542e-ffb0-4dc5-b770-85d3205df3de/volumes" Nov 01 09:58:45.479812 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:58:45.479776 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:58:45.764868 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:45.764773 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:58:52.957028 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:52.956957 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:58:52.999764 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:52.999707 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb"] Nov 01 09:58:52.999977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:52.999930 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" podUID="6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" containerName="maas-api" containerID="cri-o://dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18" gracePeriod=30 Nov 01 09:58:53.080381 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.080334 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh\" (UniqueName: \"kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh\") pod \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " Nov 01 09:58:53.080594 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.080443 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume\") pod \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\" (UID: \"e555ea93-40d7-44ca-82b5-be5e1d2624c6\") " Nov 01 09:58:53.080862 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.080836 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume" pod "e555ea93-40d7-44ca-82b5-be5e1d2624c6" (UID: "e555ea93-40d7-44ca-82b5-be5e1d2624c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:58:53.081160 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.081138 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e555ea93-40d7-44ca-82b5-be5e1d2624c6-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:53.082732 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.082695 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh" pod "e555ea93-40d7-44ca-82b5-be5e1d2624c6" (UID: "e555ea93-40d7-44ca-82b5-be5e1d2624c6"). InnerVolumeSpecName "kube-api-access-m2jjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:58:53.183009 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.182970 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jjh\" (UniqueName: \"kubernetes.io/projected/e555ea93-40d7-44ca-82b5-be5e1d2624c6-kube-api-access-m2jjh\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:53.296449 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.296423 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:58:53.385426 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.385391 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs\") pod \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " Nov 01 09:58:53.385604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.385459 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca\") pod \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " Nov 01 09:58:53.385604 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.385505 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv\" (UniqueName: \"kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv\") pod \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " Nov 01 09:58:53.385690 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.385668 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\") pod \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\" (UID: \"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd\") " Nov 01 09:58:53.385926 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.385896 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca" pod "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" (UID: "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:58:53.386181 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.386159 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:53.387889 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.387867 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls" pod "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" (UID: "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:58:53.387977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.387887 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs" pod "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" (UID: "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:58:53.387977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.387945 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv" pod "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" (UID: "6d1ad7a2-34e4-4fd3-9841-a7059bd752bd"). InnerVolumeSpecName "kube-api-access-wdkwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:58:53.487500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.487459 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:53.487500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.487494 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdkwv\" (UniqueName: \"kubernetes.io/projected/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-kube-api-access-wdkwv\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:53.487500 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.487507 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:58:53.747472 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.747436 5710 generic.go:356] "Generic (PLEG): container finished" podID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerID="73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80" exitCode=0 Nov 01 09:58:53.747699 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.747532 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" Nov 01 09:58:53.747699 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.747530 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" event={"ID":"e555ea93-40d7-44ca-82b5-be5e1d2624c6","Type":"ContainerDied","Data":"73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80"} Nov 01 09:58:53.747699 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.747693 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp" event={"ID":"e555ea93-40d7-44ca-82b5-be5e1d2624c6","Type":"ContainerDied","Data":"bc7735bbb28213d7c173049452520e4a9094c3a10da1294ebaab818aa7656a70"} Nov 01 09:58:53.747900 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.747748 5710 scope.go:122] "RemoveContainer" containerID="73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80" Nov 01 09:58:53.749103 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.749081 5710 generic.go:356] "Generic (PLEG): container finished" podID="6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" containerID="dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18" exitCode=0 Nov 01 09:58:53.749213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.749143 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" Nov 01 09:58:53.749213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.749167 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" event={"ID":"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd","Type":"ContainerDied","Data":"dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18"} Nov 01 09:58:53.749213 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.749211 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb" event={"ID":"6d1ad7a2-34e4-4fd3-9841-a7059bd752bd","Type":"ContainerDied","Data":"81f449f64fc87b3eed760df22674c4e064e663b09c2d18f62eb72786e9f88aef"} Nov 01 09:58:53.759179 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.759156 5710 scope.go:122] "RemoveContainer" containerID="73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80" Nov 01 09:58:53.759522 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:58:53.759500 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80\": container with ID starting with 73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80 not found: ID does not exist" containerID="73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80" Nov 01 09:58:53.759610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.759531 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80"} err="failed to get container status \"73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80\": rpc error: code = NotFound desc = could not find container \"73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80\": container with ID starting with 73daea53bf2f331f187c609c62a46ebcea8c5b3bedee21e27f2804d5f673cb80 not found: ID does not exist" Nov 01 09:58:53.759610 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.759552 5710 scope.go:122] "RemoveContainer" containerID="dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18" Nov 01 09:58:53.769555 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.769536 5710 scope.go:122] "RemoveContainer" containerID="dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18" Nov 01 09:58:53.769855 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:58:53.769834 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18\": container with ID starting with dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18 not found: ID does not exist" containerID="dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18" Nov 01 09:58:53.769906 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.769869 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18"} err="failed to get container status \"dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18\": rpc error: code = NotFound desc = could not find container \"dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18\": container with ID starting with dccbe749d3f9626bc86a29c54db552940a2fb234ac2a9fd5a60e8d3b0d181f18 not found: ID does not exist" Nov 01 09:58:53.903308 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.903269 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp"] Nov 01 09:58:53.913586 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:53.913532 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fd5a10-bccf4699-r4mzp"] Nov 01 09:58:54.015452 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:54.015355 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb"] Nov 01 09:58:54.023329 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:54.023299 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fd5a10-54c8dc7b4f-j6bbb"] Nov 01 09:58:55.059292 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:55.059240 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" path="/var/lib/kubelet/pods/6d1ad7a2-34e4-4fd3-9841-a7059bd752bd/volumes" Nov 01 09:58:55.059769 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:55.059687 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" path="/var/lib/kubelet/pods/e555ea93-40d7-44ca-82b5-be5e1d2624c6/volumes" Nov 01 09:58:56.840602 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:58:56.840560 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:59:01.683382 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:01.683339 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc"] Nov 01 09:59:01.683922 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:01.683600 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" containerID="cri-o://f5677dbfaadd1b2f55b984e4de1b257c7d902f7c33ab661d65b62ad2a35cb4d8" gracePeriod=30 Nov 01 09:59:06.877478 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:06.877445 5710 generic.go:356] "Generic (PLEG): container finished" podID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerID="f5677dbfaadd1b2f55b984e4de1b257c7d902f7c33ab661d65b62ad2a35cb4d8" exitCode=0 Nov 01 09:59:06.877976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:06.877543 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" event={"ID":"ed492f1a-d299-43d0-a059-af6d93321d0e","Type":"ContainerDied","Data":"f5677dbfaadd1b2f55b984e4de1b257c7d902f7c33ab661d65b62ad2a35cb4d8"} Nov 01 09:59:06.877976 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:06.877591 5710 scope.go:122] "RemoveContainer" containerID="d40e5ba0cd09d18a472146ea2184ab0b7cda6415774f8412057c5284cb5e6cb3" Nov 01 09:59:06.950673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:06.950610 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:59:07.067399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067365 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067400 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067429 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067469 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067662 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067494 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067704 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:07.067855 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067750 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067860 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:07.067977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067869 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:07.067977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067875 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.067977 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.067928 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.068208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068004 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket\") pod \"ed492f1a-d299-43d0-a059-af6d93321d0e\" (UID: \"ed492f1a-d299-43d0-a059-af6d93321d0e\") " Nov 01 09:59:07.068208 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068180 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:59:07.068508 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068372 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:07.068821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068689 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.068956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068854 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.068956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068870 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed492f1a-d299-43d0-a059-af6d93321d0e-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.068956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068884 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.068956 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.068898 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.070092 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.070067 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:07.070321 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.070298 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:07.070399 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.070355 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "kube-api-access-944fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:07.070479 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.070456 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo" pod "ed492f1a-d299-43d0-a059-af6d93321d0e" (UID: "ed492f1a-d299-43d0-a059-af6d93321d0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:59:07.169559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.169528 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.169559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.169556 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.169559 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.169565 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.169829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.169574 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-944fr\" (UniqueName: \"kubernetes.io/projected/ed492f1a-d299-43d0-a059-af6d93321d0e-kube-api-access-944fr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:07.598490 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.598455 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp"] Nov 01 09:59:07.703327 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.703288 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq"] Nov 01 09:59:07.885212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.885126 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" Nov 01 09:59:07.885212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.885157 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc" event={"ID":"ed492f1a-d299-43d0-a059-af6d93321d0e","Type":"ContainerDied","Data":"c7870563aa78c447c5de37cf2d398d417b8fe65253ce4849e69f699423873c32"} Nov 01 09:59:07.885709 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:07.885218 5710 scope.go:122] "RemoveContainer" containerID="f5677dbfaadd1b2f55b984e4de1b257c7d902f7c33ab661d65b62ad2a35cb4d8" Nov 01 09:59:08.041109 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:08.041056 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc"] Nov 01 09:59:08.045829 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:08.045787 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fd5a10-openshift-default-5544b8d7c-jzjgc"] Nov 01 09:59:09.058410 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:09.058376 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" path="/var/lib/kubelet/pods/ed492f1a-d299-43d0-a059-af6d93321d0e/volumes" Nov 01 09:59:09.312984 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:09.312874 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:59:12.599093 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:12.599050 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerName="payload-processing" containerID="cri-o://04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b" gracePeriod=25 Nov 01 09:59:12.703821 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:12.703742 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerName="payload-pre-processing" containerID="cri-o://076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180" gracePeriod=25 Nov 01 09:59:15.478046 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:59:15.478001 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:59:18.121990 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:18.121949 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:59:19.129070 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:19.129026 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:59:24.253041 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.253004 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq"] Nov 01 09:59:24.253488 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.253228 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" podUID="f8adc733-8842-4850-a899-a95975c783e5" containerName="maas-api" containerID="cri-o://782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9" gracePeriod=30 Nov 01 09:59:24.549127 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.549099 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:59:24.695054 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.695015 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p\" (UniqueName: \"kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p\") pod \"f8adc733-8842-4850-a899-a95975c783e5\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " Nov 01 09:59:24.695461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.695430 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca\") pod \"f8adc733-8842-4850-a899-a95975c783e5\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " Nov 01 09:59:24.695606 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.695529 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\") pod \"f8adc733-8842-4850-a899-a95975c783e5\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " Nov 01 09:59:24.695673 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.695613 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs\") pod \"f8adc733-8842-4850-a899-a95975c783e5\" (UID: \"f8adc733-8842-4850-a899-a95975c783e5\") " Nov 01 09:59:24.696420 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.696387 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca" pod "f8adc733-8842-4850-a899-a95975c783e5" (UID: "f8adc733-8842-4850-a899-a95975c783e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:59:24.697920 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.697890 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls" pod "f8adc733-8842-4850-a899-a95975c783e5" (UID: "f8adc733-8842-4850-a899-a95975c783e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:59:24.698119 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.698093 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p" pod "f8adc733-8842-4850-a899-a95975c783e5" (UID: "f8adc733-8842-4850-a899-a95975c783e5"). InnerVolumeSpecName "kube-api-access-g4m2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:24.698212 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.698192 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs" pod "f8adc733-8842-4850-a899-a95975c783e5" (UID: "f8adc733-8842-4850-a899-a95975c783e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 09:59:24.797944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.797845 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8adc733-8842-4850-a899-a95975c783e5-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:24.797944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.797888 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:24.797944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.797902 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8adc733-8842-4850-a899-a95975c783e5-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:24.797944 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.797915 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4m2p\" (UniqueName: \"kubernetes.io/projected/f8adc733-8842-4850-a899-a95975c783e5-kube-api-access-g4m2p\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:24.997951 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.997911 5710 generic.go:356] "Generic (PLEG): container finished" podID="f8adc733-8842-4850-a899-a95975c783e5" containerID="782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9" exitCode=0 Nov 01 09:59:24.998167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.997957 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" Nov 01 09:59:24.998167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.998007 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" event={"ID":"f8adc733-8842-4850-a899-a95975c783e5","Type":"ContainerDied","Data":"782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9"} Nov 01 09:59:24.998167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.998068 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq" event={"ID":"f8adc733-8842-4850-a899-a95975c783e5","Type":"ContainerDied","Data":"29727cc62c31d141e0a7020f5b0c7244d9ab6714e9e24ba0db69e6ccf9e587f5"} Nov 01 09:59:24.998167 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:24.998095 5710 scope.go:122] "RemoveContainer" containerID="782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9" Nov 01 09:59:25.008972 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:25.008953 5710 scope.go:122] "RemoveContainer" containerID="782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9" Nov 01 09:59:25.009286 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:59:25.009258 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9\": container with ID starting with 782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9 not found: ID does not exist" containerID="782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9" Nov 01 09:59:25.009391 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:25.009297 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9"} err="failed to get container status \"782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9\": rpc error: code = NotFound desc = could not find container \"782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9\": container with ID starting with 782f505121249a4d66701082ccaf9a6faba415e3c0b979329adc7bb1c077c3a9 not found: ID does not exist" Nov 01 09:59:25.154367 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:25.154334 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq"] Nov 01 09:59:25.156262 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:25.156235 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf99-6cfd565f6f-646rq"] Nov 01 09:59:27.058633 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:27.058598 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8adc733-8842-4850-a899-a95975c783e5" path="/var/lib/kubelet/pods/f8adc733-8842-4850-a899-a95975c783e5/volumes" Nov 01 09:59:28.122471 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:28.122432 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:59:29.129124 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:29.129082 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 09:59:30.752332 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:30.752300 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 09:59:32.355276 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.355218 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv"] Nov 01 09:59:32.355840 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.355459 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" containerID="cri-o://eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742" gracePeriod=30 Nov 01 09:59:32.862374 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.862344 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:59:32.900039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.899494 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp\" (UniqueName: \"kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp\") pod \"635db112-2d4d-436d-9ceb-9a7e80314b4e\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " Nov 01 09:59:32.900039 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.899587 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume\") pod \"635db112-2d4d-436d-9ceb-9a7e80314b4e\" (UID: \"635db112-2d4d-436d-9ceb-9a7e80314b4e\") " Nov 01 09:59:32.900689 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.900654 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume" pod "635db112-2d4d-436d-9ceb-9a7e80314b4e" (UID: "635db112-2d4d-436d-9ceb-9a7e80314b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:59:32.902844 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.902808 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp" pod "635db112-2d4d-436d-9ceb-9a7e80314b4e" (UID: "635db112-2d4d-436d-9ceb-9a7e80314b4e"). InnerVolumeSpecName "kube-api-access-mzhgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:32.951192 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:32.951166 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:59:33.001030 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.000978 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv\" (UniqueName: \"kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv\") pod \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " Nov 01 09:59:33.001260 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.001051 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume\") pod \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\" (UID: \"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49\") " Nov 01 09:59:33.001400 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.001379 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzhgp\" (UniqueName: \"kubernetes.io/projected/635db112-2d4d-436d-9ceb-9a7e80314b4e-kube-api-access-mzhgp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:33.001400 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.001398 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635db112-2d4d-436d-9ceb-9a7e80314b4e-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:33.001575 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.001543 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume" pod "6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" (UID: "6d4a42e2-3ca9-4fd5-85cc-97760d63ca49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:59:33.003456 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.003432 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv" pod "6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" (UID: "6d4a42e2-3ca9-4fd5-85cc-97760d63ca49"). InnerVolumeSpecName "kube-api-access-rlwsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:33.044554 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.044519 5710 generic.go:356] "Generic (PLEG): container finished" podID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerID="076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180" exitCode=0 Nov 01 09:59:33.044786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.044570 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" Nov 01 09:59:33.044786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.044607 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" event={"ID":"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49","Type":"ContainerDied","Data":"076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180"} Nov 01 09:59:33.044786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.044708 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq" event={"ID":"6d4a42e2-3ca9-4fd5-85cc-97760d63ca49","Type":"ContainerDied","Data":"9d00b9f181579e385db1d4a81cea749e08342fd40a0b6afe9e07be902b98beec"} Nov 01 09:59:33.044786 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.044744 5710 scope.go:122] "RemoveContainer" containerID="076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180" Nov 01 09:59:33.046265 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.046238 5710 generic.go:356] "Generic (PLEG): container finished" podID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerID="04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b" exitCode=0 Nov 01 09:59:33.046388 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.046290 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" Nov 01 09:59:33.046448 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.046294 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" event={"ID":"635db112-2d4d-436d-9ceb-9a7e80314b4e","Type":"ContainerDied","Data":"04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b"} Nov 01 09:59:33.046497 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.046445 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp" event={"ID":"635db112-2d4d-436d-9ceb-9a7e80314b4e","Type":"ContainerDied","Data":"73afd9d3b8409f3fc91c0b630d31e131e07a96169cc6bd906a5ccb406091fb56"} Nov 01 09:59:33.056351 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.056325 5710 scope.go:122] "RemoveContainer" containerID="076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180" Nov 01 09:59:33.056813 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:59:33.056670 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180\": container with ID starting with 076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180 not found: ID does not exist" containerID="076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180" Nov 01 09:59:33.056813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.056705 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180"} err="failed to get container status \"076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180\": rpc error: code = NotFound desc = could not find container \"076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180\": container with ID starting with 076227a006fca1d09b2a68553077d722218b32e8123cc42a5440dad8b9e97180 not found: ID does not exist" Nov 01 09:59:33.056813 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.056749 5710 scope.go:122] "RemoveContainer" containerID="04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b" Nov 01 09:59:33.066443 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.066420 5710 scope.go:122] "RemoveContainer" containerID="04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b" Nov 01 09:59:33.066805 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:59:33.066774 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b\": container with ID starting with 04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b not found: ID does not exist" containerID="04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b" Nov 01 09:59:33.066914 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.066809 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b"} err="failed to get container status \"04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b\": rpc error: code = NotFound desc = could not find container \"04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b\": container with ID starting with 04403aa7ad70c6d12e1e9e3b3700116931bae16b3af1e88b16b06e8559322e1b not found: ID does not exist" Nov 01 09:59:33.102522 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.102476 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlwsv\" (UniqueName: \"kubernetes.io/projected/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-kube-api-access-rlwsv\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:33.102522 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.102515 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:33.184781 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.184686 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp"] Nov 01 09:59:33.193461 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.193423 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf99-5858d4dc95-9pjdp"] Nov 01 09:59:33.298789 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.298750 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq"] Nov 01 09:59:33.304975 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:33.304941 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf99-7c8df886c5-gp8mq"] Nov 01 09:59:35.057961 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:35.057925 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" path="/var/lib/kubelet/pods/635db112-2d4d-436d-9ceb-9a7e80314b4e/volumes" Nov 01 09:59:35.058512 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:35.058492 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" path="/var/lib/kubelet/pods/6d4a42e2-3ca9-4fd5-85cc-97760d63ca49/volumes" Nov 01 09:59:37.631152 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.631081 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:59:37.657416 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657376 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.657605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657422 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.657605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657473 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.657605 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657505 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.657811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657648 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.657811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657681 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.657811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657742 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:37.657811 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657804 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.658027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657820 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:37.658027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657850 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.658027 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.657900 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data\") pod \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\" (UID: \"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05\") " Nov 01 09:59:37.658203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658106 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:37.658203 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658142 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 09:59:37.658377 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658307 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:37.658562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658477 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.658562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658491 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.658562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658503 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.658562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658516 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.658562 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.658525 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.660081 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.660057 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 09:59:37.660251 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.660231 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:37.660556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.660526 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 09:59:37.660556 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.660544 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5" pod "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" (UID: "45bc6462-8a3d-4b79-b3b1-cef9a1a93e05"). InnerVolumeSpecName "kube-api-access-8vxg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 09:59:37.759242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.759199 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxg5\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-kube-api-access-8vxg5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.759242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.759238 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.759242 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.759265 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:37.759523 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:37.759276 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 09:59:38.077320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.077220 5710 generic.go:356] "Generic (PLEG): container finished" podID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerID="eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742" exitCode=0 Nov 01 09:59:38.077320 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.077277 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" Nov 01 09:59:38.077581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.077310 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" event={"ID":"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05","Type":"ContainerDied","Data":"eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742"} Nov 01 09:59:38.077581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.077384 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv" event={"ID":"45bc6462-8a3d-4b79-b3b1-cef9a1a93e05","Type":"ContainerDied","Data":"04511e3cc1cc7176fd408a4b0b38daf163d48404dba80a7135a3e567195bfb5e"} Nov 01 09:59:38.077581 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.077413 5710 scope.go:122] "RemoveContainer" containerID="eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742" Nov 01 09:59:38.094198 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.094167 5710 scope.go:122] "RemoveContainer" containerID="eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742" Nov 01 09:59:38.094583 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:59:38.094561 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742\": container with ID starting with eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742 not found: ID does not exist" containerID="eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742" Nov 01 09:59:38.094665 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.094593 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742"} err="failed to get container status \"eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742\": rpc error: code = NotFound desc = could not find container \"eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742\": container with ID starting with eb9a81a2a81cff04a01a7eca5e1fa44f4718527768b7721d58775f01e6cde742 not found: ID does not exist" Nov 01 09:59:38.233059 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.233016 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv"] Nov 01 09:59:38.279139 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:38.279094 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf99-openshift-default-7f4d764597-vmfjv"] Nov 01 09:59:39.057878 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:39.057844 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" path="/var/lib/kubelet/pods/45bc6462-8a3d-4b79-b3b1-cef9a1a93e05/volumes" Nov 01 09:59:45.476492 ip-10-0-143-16 kubenswrapper[5710]: E1101 09:59:45.476451 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 09:59:52.987385 ip-10-0-143-16 kubenswrapper[5710]: I1101 09:59:52.987342 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:00:15.480987 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:00:15.480941 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:00:20.821823 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:20.821787 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt"] Nov 01 10:00:20.822250 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:20.822021 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" containerID="cri-o://6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905" gracePeriod=30 Nov 01 10:00:25.546400 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:25.546349 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:00:26.094813 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.094741 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 10:00:26.126984 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.126945 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127188 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.126998 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127188 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127032 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127188 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127065 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127188 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127093 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127201 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127271 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127320 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127351 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo\") pod \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\" (UID: \"f5e899f4-3ccc-4676-b68b-f9272cedb9b8\") " Nov 01 10:00:26.127461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127435 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:00:26.127761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127456 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:00:26.127761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127559 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:00:26.127761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127741 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:00:26.127761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.127744 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:00:26.128070 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.128049 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.128112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.128076 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.128112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.128092 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.128112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.128105 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.128213 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.128119 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.129624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.129590 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:00:26.129624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.129600 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "kube-api-access-gp7bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:00:26.129890 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.129624 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 10:00:26.130004 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.129980 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token" pod "f5e899f4-3ccc-4676-b68b-f9272cedb9b8" (UID: "f5e899f4-3ccc-4676-b68b-f9272cedb9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:00:26.229571 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.229535 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.229571 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.229565 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7bp\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-kube-api-access-gp7bp\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.229571 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.229576 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.229887 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.229585 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e899f4-3ccc-4676-b68b-f9272cedb9b8-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:00:26.362772 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.362637 5710 generic.go:356] "Generic (PLEG): container finished" podID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerID="6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905" exitCode=0 Nov 01 10:00:26.362772 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.362688 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" Nov 01 10:00:26.362772 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.362757 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" event={"ID":"f5e899f4-3ccc-4676-b68b-f9272cedb9b8","Type":"ContainerDied","Data":"6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905"} Nov 01 10:00:26.363043 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.362815 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt" event={"ID":"f5e899f4-3ccc-4676-b68b-f9272cedb9b8","Type":"ContainerDied","Data":"15b9d4d4e7e110b67c3603058bd313f3be37948801af2c571c7eb88c1ea73eb3"} Nov 01 10:00:26.363043 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.362841 5710 scope.go:122] "RemoveContainer" containerID="6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905" Nov 01 10:00:26.374312 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.374283 5710 scope.go:122] "RemoveContainer" containerID="6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905" Nov 01 10:00:26.374701 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:00:26.374679 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905\": container with ID starting with 6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905 not found: ID does not exist" containerID="6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905" Nov 01 10:00:26.374793 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.374736 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905"} err="failed to get container status \"6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905\": rpc error: code = NotFound desc = could not find container \"6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905\": container with ID starting with 6248977d6617928656b8611d358ec17d96273528ea2804cb71fc87aaef817905 not found: ID does not exist" Nov 01 10:00:26.519207 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.519165 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt"] Nov 01 10:00:26.522928 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:26.522898 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b711ae-openshift-default-79479496c5-n9pjt"] Nov 01 10:00:27.059072 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:27.059033 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" path="/var/lib/kubelet/pods/f5e899f4-3ccc-4676-b68b-f9272cedb9b8/volumes" Nov 01 10:00:45.477205 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:00:45.477166 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:00:51.788754 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:00:51.788697 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:01:04.674370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:04.674319 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:01:15.481974 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:01:15.481932 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:01:19.655270 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:19.655227 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4"] Nov 01 10:01:19.655846 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:19.655498 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" podUID="1a24441e-066c-43b0-b4ea-509691a6f607" containerName="maas-api" containerID="cri-o://f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e" gracePeriod=30 Nov 01 10:01:19.944709 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:19.944675 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 10:01:20.032770 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.032704 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca\") pod \"1a24441e-066c-43b0-b4ea-509691a6f607\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " Nov 01 10:01:20.033171 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.033148 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs\") pod \"1a24441e-066c-43b0-b4ea-509691a6f607\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " Nov 01 10:01:20.033397 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.033154 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca" pod "1a24441e-066c-43b0-b4ea-509691a6f607" (UID: "1a24441e-066c-43b0-b4ea-509691a6f607"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:01:20.033663 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.033643 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh\" (UniqueName: \"kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh\") pod \"1a24441e-066c-43b0-b4ea-509691a6f607\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " Nov 01 10:01:20.034551 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.034530 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\") pod \"1a24441e-066c-43b0-b4ea-509691a6f607\" (UID: \"1a24441e-066c-43b0-b4ea-509691a6f607\") " Nov 01 10:01:20.035526 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.035497 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a24441e-066c-43b0-b4ea-509691a6f607-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:20.037633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.037243 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls" pod "1a24441e-066c-43b0-b4ea-509691a6f607" (UID: "1a24441e-066c-43b0-b4ea-509691a6f607"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:01:20.037633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.037413 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs" pod "1a24441e-066c-43b0-b4ea-509691a6f607" (UID: "1a24441e-066c-43b0-b4ea-509691a6f607"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:01:20.037633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.037558 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh" pod "1a24441e-066c-43b0-b4ea-509691a6f607" (UID: "1a24441e-066c-43b0-b4ea-509691a6f607"). InnerVolumeSpecName "kube-api-access-sccxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:01:20.136377 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.136335 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sccxh\" (UniqueName: \"kubernetes.io/projected/1a24441e-066c-43b0-b4ea-509691a6f607-kube-api-access-sccxh\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:20.136377 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.136372 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:20.136377 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.136382 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a24441e-066c-43b0-b4ea-509691a6f607-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:20.657906 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.657871 5710 generic.go:356] "Generic (PLEG): container finished" podID="1a24441e-066c-43b0-b4ea-509691a6f607" containerID="f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e" exitCode=0 Nov 01 10:01:20.658447 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.657921 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" Nov 01 10:01:20.658447 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.657964 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" event={"ID":"1a24441e-066c-43b0-b4ea-509691a6f607","Type":"ContainerDied","Data":"f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e"} Nov 01 10:01:20.658447 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.658024 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4" event={"ID":"1a24441e-066c-43b0-b4ea-509691a6f607","Type":"ContainerDied","Data":"7dd2205b54b3d8c1e6c1e6ac9f7e84ebeda24fd99f787911bc4f2c5e5d2a6794"} Nov 01 10:01:20.658447 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.658052 5710 scope.go:122] "RemoveContainer" containerID="f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e" Nov 01 10:01:20.669697 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.669675 5710 scope.go:122] "RemoveContainer" containerID="f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e" Nov 01 10:01:20.670089 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:01:20.670068 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e\": container with ID starting with f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e not found: ID does not exist" containerID="f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e" Nov 01 10:01:20.670163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.670099 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e"} err="failed to get container status \"f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e\": rpc error: code = NotFound desc = could not find container \"f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e\": container with ID starting with f1317b0e7700dcf4849b08ff8024c829bbdcdace46cd7028d5480a5c2818c69e not found: ID does not exist" Nov 01 10:01:20.812967 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.812926 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4"] Nov 01 10:01:20.816832 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:20.816801 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a87a1d-bfb5fd7df-shbj4"] Nov 01 10:01:21.057549 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:21.057464 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a24441e-066c-43b0-b4ea-509691a6f607" path="/var/lib/kubelet/pods/1a24441e-066c-43b0-b4ea-509691a6f607/volumes" Nov 01 10:01:25.345711 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:25.345662 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:01:28.730785 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:28.730712 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq"] Nov 01 10:01:28.731284 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:28.730976 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" containerID="cri-o://8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf" gracePeriod=30 Nov 01 10:01:34.003327 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.003294 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 10:01:34.083416 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083320 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083416 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083368 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083495 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083515 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083555 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083586 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083623 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.083683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083666 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.084074 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083677 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:34.084074 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083706 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket\") pod \"a4ced5b6-3157-4b50-ade2-102964383121\" (UID: \"a4ced5b6-3157-4b50-ade2-102964383121\") " Nov 01 10:01:34.084074 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.083846 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:34.084074 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084050 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:34.084461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084070 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:34.084461 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084242 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:01:34.084796 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084771 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.084926 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084798 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.084926 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084815 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.084926 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.084829 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.086177 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.086151 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 10:01:34.086305 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.086198 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:01:34.086305 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.086217 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "kube-api-access-lhwzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:01:34.086305 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.086276 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy" pod "a4ced5b6-3157-4b50-ade2-102964383121" (UID: "a4ced5b6-3157-4b50-ade2-102964383121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:34.185503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.185425 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhwzn\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-kube-api-access-lhwzn\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.185503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.185458 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ced5b6-3157-4b50-ade2-102964383121-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.185503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.185468 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ced5b6-3157-4b50-ade2-102964383121-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.185503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.185478 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ced5b6-3157-4b50-ade2-102964383121-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.185503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.185487 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ced5b6-3157-4b50-ade2-102964383121-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:34.743920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.743886 5710 generic.go:356] "Generic (PLEG): container finished" podID="a4ced5b6-3157-4b50-ade2-102964383121" containerID="8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf" exitCode=0 Nov 01 10:01:34.744140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.743936 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" Nov 01 10:01:34.744140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.743962 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" event={"ID":"a4ced5b6-3157-4b50-ade2-102964383121","Type":"ContainerDied","Data":"8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf"} Nov 01 10:01:34.744140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.744018 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq" event={"ID":"a4ced5b6-3157-4b50-ade2-102964383121","Type":"ContainerDied","Data":"8567812e6432b8e2e98ed04f2aa37f941f1f8c946802a5ebd0dcb35bba4720d4"} Nov 01 10:01:34.744140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.744042 5710 scope.go:122] "RemoveContainer" containerID="8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf" Nov 01 10:01:34.754550 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.754527 5710 scope.go:122] "RemoveContainer" containerID="8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf" Nov 01 10:01:34.755053 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:01:34.755033 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf\": container with ID starting with 8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf not found: ID does not exist" containerID="8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf" Nov 01 10:01:34.755125 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.755067 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf"} err="failed to get container status \"8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf\": rpc error: code = NotFound desc = could not find container \"8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf\": container with ID starting with 8f4db3d03840d514bd9d4ee2edaa173db785e5a88273ef47ee12663b6de9ecdf not found: ID does not exist" Nov 01 10:01:34.946406 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.946365 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq"] Nov 01 10:01:34.992565 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:34.992524 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a87a1d-openshift-default-579cd9875c-hvdfq"] Nov 01 10:01:35.005627 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:35.005535 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4"] Nov 01 10:01:35.058119 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:35.058080 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ced5b6-3157-4b50-ade2-102964383121" path="/var/lib/kubelet/pods/a4ced5b6-3157-4b50-ade2-102964383121/volumes" Nov 01 10:01:35.886702 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:35.886661 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4"] Nov 01 10:01:35.886946 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:35.886908 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" containerName="maas-api" containerID="cri-o://e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa" gracePeriod=30 Nov 01 10:01:36.178059 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.178032 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 10:01:36.214884 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.214845 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2\" (UniqueName: \"kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2\") pod \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " Nov 01 10:01:36.215078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.214920 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs\") pod \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " Nov 01 10:01:36.215078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.214983 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\") pod \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " Nov 01 10:01:36.215078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.215035 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca\") pod \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\" (UID: \"380d0df9-c4c5-4912-8a44-1cb19a3d8872\") " Nov 01 10:01:36.215454 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.215426 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca" pod "380d0df9-c4c5-4912-8a44-1cb19a3d8872" (UID: "380d0df9-c4c5-4912-8a44-1cb19a3d8872"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:01:36.217352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.217323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls" pod "380d0df9-c4c5-4912-8a44-1cb19a3d8872" (UID: "380d0df9-c4c5-4912-8a44-1cb19a3d8872"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:01:36.217479 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.217375 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs" pod "380d0df9-c4c5-4912-8a44-1cb19a3d8872" (UID: "380d0df9-c4c5-4912-8a44-1cb19a3d8872"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:01:36.217479 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.217413 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2" pod "380d0df9-c4c5-4912-8a44-1cb19a3d8872" (UID: "380d0df9-c4c5-4912-8a44-1cb19a3d8872"). InnerVolumeSpecName "kube-api-access-tlqs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:01:36.316529 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.316494 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:36.316529 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.316527 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380d0df9-c4c5-4912-8a44-1cb19a3d8872-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:36.316755 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.316537 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlqs2\" (UniqueName: \"kubernetes.io/projected/380d0df9-c4c5-4912-8a44-1cb19a3d8872-kube-api-access-tlqs2\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:36.316755 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.316548 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380d0df9-c4c5-4912-8a44-1cb19a3d8872-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:36.758390 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.758357 5710 generic.go:356] "Generic (PLEG): container finished" podID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" containerID="e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa" exitCode=0 Nov 01 10:01:36.758635 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.758411 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" Nov 01 10:01:36.758635 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.758411 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" event={"ID":"380d0df9-c4c5-4912-8a44-1cb19a3d8872","Type":"ContainerDied","Data":"e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa"} Nov 01 10:01:36.758635 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.758533 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4" event={"ID":"380d0df9-c4c5-4912-8a44-1cb19a3d8872","Type":"ContainerDied","Data":"1881f8579c26d022cfca6a38f30c550ec4b1330503644a30b3db5f70a454ec0c"} Nov 01 10:01:36.758635 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.758554 5710 scope.go:122] "RemoveContainer" containerID="e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa" Nov 01 10:01:36.770113 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.770088 5710 scope.go:122] "RemoveContainer" containerID="e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa" Nov 01 10:01:36.770466 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:01:36.770441 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa\": container with ID starting with e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa not found: ID does not exist" containerID="e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa" Nov 01 10:01:36.770578 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.770477 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa"} err="failed to get container status \"e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa\": rpc error: code = NotFound desc = could not find container \"e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa\": container with ID starting with e3f2a55bdbd5b04b121d35548b4a2d585d63414e7eb59e9dc9c4f53f86003dfa not found: ID does not exist" Nov 01 10:01:36.914015 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.913973 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4"] Nov 01 10:01:36.917643 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:36.917384 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-65aeef-5c6b665788-8cfh4"] Nov 01 10:01:37.057169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:37.057085 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" path="/var/lib/kubelet/pods/380d0df9-c4c5-4912-8a44-1cb19a3d8872/volumes" Nov 01 10:01:40.006095 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:40.006053 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerName="payload-processing" containerID="cri-o://44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff" gracePeriod=25 Nov 01 10:01:41.286266 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:41.286228 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:01:44.662711 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:44.662669 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb"] Nov 01 10:01:44.663167 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:44.662915 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" containerID="cri-o://280dd54cccfba196abf4a7311c2ce31039597f9fba59511445d235ee02ba98f5" gracePeriod=30 Nov 01 10:01:45.479289 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:01:45.479251 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:01:48.909554 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:48.909505 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:01:49.829984 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:49.829952 5710 generic.go:356] "Generic (PLEG): container finished" podID="9478fa58-762a-45cc-aa2f-91803a661f30" containerID="280dd54cccfba196abf4a7311c2ce31039597f9fba59511445d235ee02ba98f5" exitCode=0 Nov 01 10:01:49.830172 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:49.830025 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" event={"ID":"9478fa58-762a-45cc-aa2f-91803a661f30","Type":"ContainerDied","Data":"280dd54cccfba196abf4a7311c2ce31039597f9fba59511445d235ee02ba98f5"} Nov 01 10:01:49.830172 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:49.830085 5710 scope.go:122] "RemoveContainer" containerID="4cd84c1306ce8ebcdc041f515d4bd736e2874e5d2877a689375823ac6d588db1" Nov 01 10:01:49.926600 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:49.926572 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 10:01:50.068113 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068012 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068113 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068080 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068156 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068195 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068279 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068309 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068347 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068349 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:50.068701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068378 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:50.068701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068405 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068478 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy\") pod \"9478fa58-762a-45cc-aa2f-91803a661f30\" (UID: \"9478fa58-762a-45cc-aa2f-91803a661f30\") " Nov 01 10:01:50.068701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068558 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:50.068701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068642 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:01:50.069008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068890 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9478fa58-762a-45cc-aa2f-91803a661f30-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.069008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068910 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.069008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068924 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.069008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068942 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.069008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.068994 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:50.070858 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.070829 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:01:50.071094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.071076 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:01:50.071258 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.071240 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 10:01:50.071335 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.071308 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh" pod "9478fa58-762a-45cc-aa2f-91803a661f30" (UID: "9478fa58-762a-45cc-aa2f-91803a661f30"). InnerVolumeSpecName "kube-api-access-b5mfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:01:50.135349 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.135307 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc"] Nov 01 10:01:50.144697 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.144655 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp"] Nov 01 10:01:50.146766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.146730 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn"] Nov 01 10:01:50.170006 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.169964 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5mfh\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-kube-api-access-b5mfh\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.170006 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.170000 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9478fa58-762a-45cc-aa2f-91803a661f30-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.170006 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.170018 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.170273 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.170031 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9478fa58-762a-45cc-aa2f-91803a661f30-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.170273 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.170043 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9478fa58-762a-45cc-aa2f-91803a661f30-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:01:50.836579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.836539 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" event={"ID":"9478fa58-762a-45cc-aa2f-91803a661f30","Type":"ContainerDied","Data":"380efa718ef6138595678fcca1ff0919a4e1c3e9ebc4ad3523b97fd3dd060dc5"} Nov 01 10:01:50.836579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.836562 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb" Nov 01 10:01:50.836868 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.836604 5710 scope.go:122] "RemoveContainer" containerID="280dd54cccfba196abf4a7311c2ce31039597f9fba59511445d235ee02ba98f5" Nov 01 10:01:50.993118 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:50.993080 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb"] Nov 01 10:01:51.039421 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:51.039383 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-65aeef-openshift-default-7c588bd86c-xkljb"] Nov 01 10:01:51.057517 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:51.057487 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" path="/var/lib/kubelet/pods/9478fa58-762a-45cc-aa2f-91803a661f30/volumes" Nov 01 10:01:55.859649 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:55.859607 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:01:58.910026 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:58.909983 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:01:58.941404 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:01:58.941362 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv"] Nov 01 10:02:00.270486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.270414 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 10:02:00.373915 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.373879 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4\" (UniqueName: \"kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4\") pod \"9489fe56-209e-4fef-b29b-85e9b338ccf5\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " Nov 01 10:02:00.374108 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.374004 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume\") pod \"9489fe56-209e-4fef-b29b-85e9b338ccf5\" (UID: \"9489fe56-209e-4fef-b29b-85e9b338ccf5\") " Nov 01 10:02:00.374505 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.374475 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume" pod "9489fe56-209e-4fef-b29b-85e9b338ccf5" (UID: "9489fe56-209e-4fef-b29b-85e9b338ccf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:02:00.376224 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.376197 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4" pod "9489fe56-209e-4fef-b29b-85e9b338ccf5" (UID: "9489fe56-209e-4fef-b29b-85e9b338ccf5"). InnerVolumeSpecName "kube-api-access-n9lq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:00.475154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.475120 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9lq4\" (UniqueName: \"kubernetes.io/projected/9489fe56-209e-4fef-b29b-85e9b338ccf5-kube-api-access-n9lq4\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:00.475154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.475152 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9489fe56-209e-4fef-b29b-85e9b338ccf5-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:00.910409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.910368 5710 generic.go:356] "Generic (PLEG): container finished" podID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerID="44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff" exitCode=0 Nov 01 10:02:00.910661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.910441 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" Nov 01 10:02:00.910661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.910460 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" event={"ID":"9489fe56-209e-4fef-b29b-85e9b338ccf5","Type":"ContainerDied","Data":"44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff"} Nov 01 10:02:00.910661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.910534 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4" event={"ID":"9489fe56-209e-4fef-b29b-85e9b338ccf5","Type":"ContainerDied","Data":"fb236a58e72e3879fe2617e4433f0c6a00f9d8085e745f2a101e59fbceed95e5"} Nov 01 10:02:00.910661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.910569 5710 scope.go:122] "RemoveContainer" containerID="44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff" Nov 01 10:02:00.921308 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.921286 5710 scope.go:122] "RemoveContainer" containerID="44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff" Nov 01 10:02:00.921654 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:00.921633 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff\": container with ID starting with 44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff not found: ID does not exist" containerID="44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff" Nov 01 10:02:00.921729 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:00.921665 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff"} err="failed to get container status \"44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff\": rpc error: code = NotFound desc = could not find container \"44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff\": container with ID starting with 44f8b01149f553bc23a180e0fe5e5b596cdc09b3a4aac801c91a492c9c106bff not found: ID does not exist" Nov 01 10:02:01.063909 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:01.063868 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4"] Nov 01 10:02:01.109846 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:01.109807 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-65aeef-6988d8d5b5-pdvz4"] Nov 01 10:02:03.058727 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:03.058684 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" path="/var/lib/kubelet/pods/9489fe56-209e-4fef-b29b-85e9b338ccf5/volumes" Nov 01 10:02:03.942369 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:03.942327 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" podUID="a81fbead-001d-408c-946b-0bff9281b071" containerName="payload-pre-processing" containerID="cri-o://4ede7eaa58dc1569cb14752cfdd04191193fb75eac6a0b0286a77084580fc7b7" gracePeriod=25 Nov 01 10:02:05.159832 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.159790 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" containerID="cri-o://8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585" gracePeriod=45 Nov 01 10:02:05.169263 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.169220 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" containerID="cri-o://d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9" gracePeriod=45 Nov 01 10:02:05.169435 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.169221 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" containerID="cri-o://f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a" gracePeriod=45 Nov 01 10:02:05.482342 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.482311 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 10:02:05.486813 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.486786 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 10:02:05.492485 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.492457 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 10:02:05.532423 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532391 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir\") pod \"0805ce31-0624-45fe-a11d-31e49ec7b01b\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532430 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache\") pod \"0805ce31-0624-45fe-a11d-31e49ec7b01b\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532451 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l\" (UniqueName: \"kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l\") pod \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532496 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home\") pod \"0ef41f20-0c0c-4283-812c-8141c9b3c733\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532544 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs\") pod \"0805ce31-0624-45fe-a11d-31e49ec7b01b\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532576 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm\") pod \"0805ce31-0624-45fe-a11d-31e49ec7b01b\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532611 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5\" (UniqueName: \"kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5\") pod \"0ef41f20-0c0c-4283-812c-8141c9b3c733\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532656 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm\") pod \"0ef41f20-0c0c-4283-812c-8141c9b3c733\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532683 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir\") pod \"0ef41f20-0c0c-4283-812c-8141c9b3c733\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " Nov 01 10:02:05.532766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532751 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829\" (UniqueName: \"kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829\") pod \"0805ce31-0624-45fe-a11d-31e49ec7b01b\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532806 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs\") pod \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532842 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache\") pod \"0ef41f20-0c0c-4283-812c-8141c9b3c733\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532868 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir\") pod \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532912 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs\") pod \"0ef41f20-0c0c-4283-812c-8141c9b3c733\" (UID: \"0ef41f20-0c0c-4283-812c-8141c9b3c733\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.532952 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm\") pod \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533005 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache\") pod \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533042 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home\") pod \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\" (UID: \"b3ff0e23-0b62-4c33-97f7-14357d8675cf\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533075 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home\") pod \"0805ce31-0624-45fe-a11d-31e49ec7b01b\" (UID: \"0805ce31-0624-45fe-a11d-31e49ec7b01b\") " Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533083 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir" pod "0805ce31-0624-45fe-a11d-31e49ec7b01b" (UID: "0805ce31-0624-45fe-a11d-31e49ec7b01b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533298 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache" pod "0805ce31-0624-45fe-a11d-31e49ec7b01b" (UID: "0805ce31-0624-45fe-a11d-31e49ec7b01b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.533307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533292 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home" pod "0ef41f20-0c0c-4283-812c-8141c9b3c733" (UID: "0ef41f20-0c0c-4283-812c-8141c9b3c733"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.533930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533499 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache" pod "b3ff0e23-0b62-4c33-97f7-14357d8675cf" (UID: "b3ff0e23-0b62-4c33-97f7-14357d8675cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.533930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533641 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.533930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533661 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.533930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533676 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.533930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533690 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.534201 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533920 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home" pod "b3ff0e23-0b62-4c33-97f7-14357d8675cf" (UID: "b3ff0e23-0b62-4c33-97f7-14357d8675cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.534201 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.533999 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache" pod "0ef41f20-0c0c-4283-812c-8141c9b3c733" (UID: "0ef41f20-0c0c-4283-812c-8141c9b3c733"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.534201 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.534192 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home" pod "0805ce31-0624-45fe-a11d-31e49ec7b01b" (UID: "0805ce31-0624-45fe-a11d-31e49ec7b01b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.534989 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.534865 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir" pod "0ef41f20-0c0c-4283-812c-8141c9b3c733" (UID: "0ef41f20-0c0c-4283-812c-8141c9b3c733"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.534989 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.534954 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir" pod "b3ff0e23-0b62-4c33-97f7-14357d8675cf" (UID: "b3ff0e23-0b62-4c33-97f7-14357d8675cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.536635 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.536608 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm" pod "b3ff0e23-0b62-4c33-97f7-14357d8675cf" (UID: "b3ff0e23-0b62-4c33-97f7-14357d8675cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.537151 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.537128 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs" pod "0ef41f20-0c0c-4283-812c-8141c9b3c733" (UID: "0ef41f20-0c0c-4283-812c-8141c9b3c733"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:02:05.538017 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.537996 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829" pod "0805ce31-0624-45fe-a11d-31e49ec7b01b" (UID: "0805ce31-0624-45fe-a11d-31e49ec7b01b"). InnerVolumeSpecName "kube-api-access-lw829". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:05.538166 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.538149 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm" pod "0ef41f20-0c0c-4283-812c-8141c9b3c733" (UID: "0ef41f20-0c0c-4283-812c-8141c9b3c733"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.538233 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.538176 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs" pod "0805ce31-0624-45fe-a11d-31e49ec7b01b" (UID: "0805ce31-0624-45fe-a11d-31e49ec7b01b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:02:05.538274 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.538236 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs" pod "b3ff0e23-0b62-4c33-97f7-14357d8675cf" (UID: "b3ff0e23-0b62-4c33-97f7-14357d8675cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:02:05.538745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.538687 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5" pod "0ef41f20-0c0c-4283-812c-8141c9b3c733" (UID: "0ef41f20-0c0c-4283-812c-8141c9b3c733"). InnerVolumeSpecName "kube-api-access-trkv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:05.539196 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.539169 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l" pod "b3ff0e23-0b62-4c33-97f7-14357d8675cf" (UID: "b3ff0e23-0b62-4c33-97f7-14357d8675cf"). InnerVolumeSpecName "kube-api-access-7266l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:05.539705 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.539686 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm" pod "0805ce31-0624-45fe-a11d-31e49ec7b01b" (UID: "0805ce31-0624-45fe-a11d-31e49ec7b01b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:05.635174 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635138 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7266l\" (UniqueName: \"kubernetes.io/projected/b3ff0e23-0b62-4c33-97f7-14357d8675cf-kube-api-access-7266l\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635174 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635169 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0805ce31-0624-45fe-a11d-31e49ec7b01b-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635174 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635179 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635188 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkv5\" (UniqueName: \"kubernetes.io/projected/0ef41f20-0c0c-4283-812c-8141c9b3c733-kube-api-access-trkv5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635197 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635205 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635221 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw829\" (UniqueName: \"kubernetes.io/projected/0805ce31-0624-45fe-a11d-31e49ec7b01b-kube-api-access-lw829\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635232 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635241 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef41f20-0c0c-4283-812c-8141c9b3c733-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635252 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635261 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef41f20-0c0c-4283-812c-8141c9b3c733-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635268 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635276 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff0e23-0b62-4c33-97f7-14357d8675cf-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.635514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.635283 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0805ce31-0624-45fe-a11d-31e49ec7b01b-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:05.973960 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.973920 5710 generic.go:356] "Generic (PLEG): container finished" podID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerID="f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a" exitCode=0 Nov 01 10:02:05.974352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.974326 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" event={"ID":"0805ce31-0624-45fe-a11d-31e49ec7b01b","Type":"ContainerDied","Data":"f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a"} Nov 01 10:02:05.974510 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.974495 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" event={"ID":"0805ce31-0624-45fe-a11d-31e49ec7b01b","Type":"ContainerDied","Data":"4bffc67c8cac06555b87bb05faf164844ba556c86a2d86edbd9841a89c2c5acb"} Nov 01 10:02:05.974616 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.974605 5710 scope.go:122] "RemoveContainer" containerID="f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a" Nov 01 10:02:05.974867 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.974848 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn" Nov 01 10:02:05.989765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.986103 5710 generic.go:356] "Generic (PLEG): container finished" podID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerID="d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9" exitCode=0 Nov 01 10:02:05.989765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.986228 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" event={"ID":"0ef41f20-0c0c-4283-812c-8141c9b3c733","Type":"ContainerDied","Data":"d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9"} Nov 01 10:02:05.989765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.986264 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" event={"ID":"0ef41f20-0c0c-4283-812c-8141c9b3c733","Type":"ContainerDied","Data":"2079d8b8f16b82b6d642f21f488165a8145e5439fac0e91c90e8bf54cb646c0d"} Nov 01 10:02:05.989765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:05.986328 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" Nov 01 10:02:06.032205 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.032030 5710 scope.go:122] "RemoveContainer" containerID="f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a" Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:06.034360 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a\": container with ID starting with f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a not found: ID does not exist" containerID="f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a" Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.034412 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a"} err="failed to get container status \"f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a\": rpc error: code = NotFound desc = could not find container \"f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a\": container with ID starting with f9bbb709dc4ac1777233f204589eb63ae6c21abfc0784e4f56560d8da7a6878a not found: ID does not exist" Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.034436 5710 scope.go:122] "RemoveContainer" containerID="d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9" Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.035921 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.038941 5710 generic.go:356] "Generic (PLEG): container finished" podID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerID="8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585" exitCode=0 Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.039068 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" event={"ID":"b3ff0e23-0b62-4c33-97f7-14357d8675cf","Type":"ContainerDied","Data":"8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585"} Nov 01 10:02:06.039954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.039108 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" event={"ID":"b3ff0e23-0b62-4c33-97f7-14357d8675cf","Type":"ContainerDied","Data":"d60ca322f5c4363ae946d21469ba2403434e5490eca2e9e4ad3bf336897be145"} Nov 01 10:02:06.062938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.062319 5710 scope.go:122] "RemoveContainer" containerID="d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9" Nov 01 10:02:06.062938 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:06.062906 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9\": container with ID starting with d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9 not found: ID does not exist" containerID="d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9" Nov 01 10:02:06.063085 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.062949 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9"} err="failed to get container status \"d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9\": rpc error: code = NotFound desc = could not find container \"d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9\": container with ID starting with d6472dbea2ce2e886fd3b3679f0fc8673a3fce7abea0db7624937a02f897f5e9 not found: ID does not exist" Nov 01 10:02:06.063085 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.062976 5710 scope.go:122] "RemoveContainer" containerID="8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585" Nov 01 10:02:06.104023 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.103681 5710 scope.go:122] "RemoveContainer" containerID="8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585" Nov 01 10:02:06.104584 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:06.104544 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585\": container with ID starting with 8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585 not found: ID does not exist" containerID="8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585" Nov 01 10:02:06.104750 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.104592 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585"} err="failed to get container status \"8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585\": rpc error: code = NotFound desc = could not find container \"8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585\": container with ID starting with 8bd8e98d34c52e2dd01b8fa5a54276131c9d6b5372c5da52dd80f4bb4a876585 not found: ID does not exist" Nov 01 10:02:06.190139 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.190088 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc"] Nov 01 10:02:06.195495 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.195458 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc"] Nov 01 10:02:06.300336 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.300240 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn"] Nov 01 10:02:06.306435 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.306399 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/simulator-w1-dfdd13-kserve-75459998d6-kfzpn"] Nov 01 10:02:06.410172 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.410133 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp"] Nov 01 10:02:06.413158 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.413115 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp"] Nov 01 10:02:06.430823 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.430771 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-2-w1-dfdd13-kserve-7b496c87b5-xz2rc" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 10:02:06.477332 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:06.477295 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfdd13/distinct-w1-dfdd13-kserve-77f7b66d46-fkrkp" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 10:02:07.058354 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:07.058316 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" path="/var/lib/kubelet/pods/0805ce31-0624-45fe-a11d-31e49ec7b01b/volumes" Nov 01 10:02:07.058903 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:07.058885 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" path="/var/lib/kubelet/pods/0ef41f20-0c0c-4283-812c-8141c9b3c733/volumes" Nov 01 10:02:07.059443 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:07.059425 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" path="/var/lib/kubelet/pods/b3ff0e23-0b62-4c33-97f7-14357d8675cf/volumes" Nov 01 10:02:09.033821 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:09.033772 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" podUID="a81fbead-001d-408c-946b-0bff9281b071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:02:12.340711 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.340674 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg"] Nov 01 10:02:12.341181 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.340890 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" podUID="3b162c79-4e90-4a17-a3ca-d0996729e047" containerName="maas-api" containerID="cri-o://ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8" gracePeriod=30 Nov 01 10:02:12.617947 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.617915 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 10:02:12.711100 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.711060 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca\") pod \"3b162c79-4e90-4a17-a3ca-d0996729e047\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " Nov 01 10:02:12.711351 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.711139 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww\" (UniqueName: \"kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww\") pod \"3b162c79-4e90-4a17-a3ca-d0996729e047\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " Nov 01 10:02:12.711351 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.711266 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\") pod \"3b162c79-4e90-4a17-a3ca-d0996729e047\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " Nov 01 10:02:12.711351 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.711303 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs\") pod \"3b162c79-4e90-4a17-a3ca-d0996729e047\" (UID: \"3b162c79-4e90-4a17-a3ca-d0996729e047\") " Nov 01 10:02:12.711577 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.711549 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca" pod "3b162c79-4e90-4a17-a3ca-d0996729e047" (UID: "3b162c79-4e90-4a17-a3ca-d0996729e047"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:02:12.711761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.711705 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b162c79-4e90-4a17-a3ca-d0996729e047-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:12.713672 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.713639 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww" pod "3b162c79-4e90-4a17-a3ca-d0996729e047" (UID: "3b162c79-4e90-4a17-a3ca-d0996729e047"). InnerVolumeSpecName "kube-api-access-xb9ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:12.713817 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.713705 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs" pod "3b162c79-4e90-4a17-a3ca-d0996729e047" (UID: "3b162c79-4e90-4a17-a3ca-d0996729e047"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:02:12.713872 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.713849 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls" pod "3b162c79-4e90-4a17-a3ca-d0996729e047" (UID: "3b162c79-4e90-4a17-a3ca-d0996729e047"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:02:12.812528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.812484 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:12.812528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.812519 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b162c79-4e90-4a17-a3ca-d0996729e047-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:12.812528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:12.812530 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb9ww\" (UniqueName: \"kubernetes.io/projected/3b162c79-4e90-4a17-a3ca-d0996729e047-kube-api-access-xb9ww\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:13.079617 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.079571 5710 generic.go:356] "Generic (PLEG): container finished" podID="3b162c79-4e90-4a17-a3ca-d0996729e047" containerID="ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8" exitCode=0 Nov 01 10:02:13.079617 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.079621 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" Nov 01 10:02:13.079889 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.079650 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" event={"ID":"3b162c79-4e90-4a17-a3ca-d0996729e047","Type":"ContainerDied","Data":"ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8"} Nov 01 10:02:13.079889 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.079709 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg" event={"ID":"3b162c79-4e90-4a17-a3ca-d0996729e047","Type":"ContainerDied","Data":"7f4c8c70cdc6ec8284fdd9a70a7f75e5b8f2a094badf69f5d3f313a4a4c0bf18"} Nov 01 10:02:13.079889 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.079745 5710 scope.go:122] "RemoveContainer" containerID="ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8" Nov 01 10:02:13.089669 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.089640 5710 scope.go:122] "RemoveContainer" containerID="ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8" Nov 01 10:02:13.090063 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:13.090040 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8\": container with ID starting with ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8 not found: ID does not exist" containerID="ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8" Nov 01 10:02:13.090112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.090077 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8"} err="failed to get container status \"ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8\": rpc error: code = NotFound desc = could not find container \"ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8\": container with ID starting with ce8be98c86e2e5607555e3880c14efd5687098f0fa01ad7a45e07e00f61f83a8 not found: ID does not exist" Nov 01 10:02:13.234593 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.234555 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg"] Nov 01 10:02:13.238283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:13.238253 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfdd13-5794b776fd-dc9zg"] Nov 01 10:02:15.056472 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:15.056439 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b162c79-4e90-4a17-a3ca-d0996729e047" path="/var/lib/kubelet/pods/3b162c79-4e90-4a17-a3ca-d0996729e047/volumes" Nov 01 10:02:15.477631 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:15.477593 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:02:16.589511 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:16.589476 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:02:18.558129 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558088 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h"] Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558491 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e1131cda-c18a-4e17-a97a-771c88b65c9f" containerName="payload-pre-processing" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558503 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1131cda-c18a-4e17-a97a-771c88b65c9f" containerName="payload-pre-processing" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558514 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558518 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558528 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558533 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558542 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f8adc733-8842-4850-a899-a95975c783e5" containerName="maas-api" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558546 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8adc733-8842-4850-a899-a95975c783e5" containerName="maas-api" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558554 5710 cpu_manager.go:414] "containerMap: removing container" podUID="af82bf01-0af4-4bdd-9f81-941504980db7" containerName="maas-api" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558558 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af82bf01-0af4-4bdd-9f81-941504980db7" containerName="maas-api" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558569 5710 cpu_manager.go:414] "containerMap: removing container" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerName="payload-processing" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558574 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerName="payload-processing" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558582 5710 cpu_manager.go:414] "containerMap: removing container" podUID="28237a9a-dc8b-4c73-957f-41e47e14689c" containerName="maas-api" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558586 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28237a9a-dc8b-4c73-957f-41e47e14689c" containerName="maas-api" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558592 5710 cpu_manager.go:414] "containerMap: removing container" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" Nov 01 10:02:18.558596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558597 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558607 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558611 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558617 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558622 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558628 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558633 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558639 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558645 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558651 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558656 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558661 5710 cpu_manager.go:414] "containerMap: removing container" podUID="93e1360d-f860-472d-a024-dae5df31ed42" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558664 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e1360d-f860-472d-a024-dae5df31ed42" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558670 5710 cpu_manager.go:414] "containerMap: removing container" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558674 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558683 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558687 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558696 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1a24441e-066c-43b0-b4ea-509691a6f607" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558700 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a24441e-066c-43b0-b4ea-509691a6f607" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558706 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0b61e9c7-f611-4cfd-a401-66225c0d1d61" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558709 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b61e9c7-f611-4cfd-a401-66225c0d1d61" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558734 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558738 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558751 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3b162c79-4e90-4a17-a3ca-d0996729e047" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558755 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b162c79-4e90-4a17-a3ca-d0996729e047" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558761 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558766 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558772 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558776 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558784 5710 cpu_manager.go:414] "containerMap: removing container" podUID="5d6be58b-baed-445c-a199-aee6ae24314a" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558788 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6be58b-baed-445c-a199-aee6ae24314a" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558796 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558800 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558805 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558810 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558816 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558821 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558828 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558833 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558839 5710 cpu_manager.go:414] "containerMap: removing container" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558843 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558849 5710 cpu_manager.go:414] "containerMap: removing container" podUID="8f5dfc48-0342-4e80-90c1-5ddae3f4497e" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558855 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5dfc48-0342-4e80-90c1-5ddae3f4497e" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558863 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558867 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558874 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b5512ce5-e232-4b9b-b8f5-19b0300f2f07" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558878 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5512ce5-e232-4b9b-b8f5-19b0300f2f07" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558936 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d357506-a8cc-4c6d-bdd5-1bc71f53045c" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558945 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef41f20-0c0c-4283-812c-8141c9b3c733" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558951 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9489fe56-209e-4fef-b29b-85e9b338ccf5" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558958 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e555ea93-40d7-44ca-82b5-be5e1d2624c6" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558963 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e1360d-f860-472d-a024-dae5df31ed42" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558969 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="89009dff-0b02-4d79-96a0-ac92c3bbe881" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558975 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558982 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a24441e-066c-43b0-b4ea-509691a6f607" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558986 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b61e9c7-f611-4cfd-a401-66225c0d1d61" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558992 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1ad7a2-34e4-4fd3-9841-a7059bd752bd" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.558998 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b162c79-4e90-4a17-a3ca-d0996729e047" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559003 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559008 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ced5b6-3157-4b50-ade2-102964383121" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559013 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6be58b-baed-445c-a199-aee6ae24314a" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559017 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a8542e-ffb0-4dc5-b770-85d3205df3de" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559023 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4a42e2-3ca9-4fd5-85cc-97760d63ca49" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559028 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e899f4-3ccc-4676-b68b-f9272cedb9b8" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559032 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e69810-df19-492c-90bd-9d56ddc2d160" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559037 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="380d0df9-c4c5-4912-8a44-1cb19a3d8872" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559042 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5dfc48-0342-4e80-90c1-5ddae3f4497e" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559048 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ff0e23-0b62-4c33-97f7-14357d8675cf" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559053 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5512ce5-e232-4b9b-b8f5-19b0300f2f07" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559059 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1131cda-c18a-4e17-a97a-771c88b65c9f" containerName="payload-pre-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559064 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed492f1a-d299-43d0-a059-af6d93321d0e" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559069 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0805ce31-0624-45fe-a11d-31e49ec7b01b" containerName="main" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559075 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8adc733-8842-4850-a899-a95975c783e5" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559079 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="af82bf01-0af4-4bdd-9f81-941504980db7" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559085 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="635db112-2d4d-436d-9ceb-9a7e80314b4e" containerName="payload-processing" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559091 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="28237a9a-dc8b-4c73-957f-41e47e14689c" containerName="maas-api" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559096 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bc6462-8a3d-4b79-b3b1-cef9a1a93e05" containerName="istio-proxy" Nov 01 10:02:18.559186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.559101 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9478fa58-762a-45cc-aa2f-91803a661f30" containerName="istio-proxy" Nov 01 10:02:18.563981 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.563959 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.661624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.661590 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h"] Nov 01 10:02:18.670203 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.670165 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7d6w\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.670528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.670504 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.670709 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.670692 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.670877 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.670858 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.671002 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.670988 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.671131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.671118 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.671249 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.671234 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.671352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.671337 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.671473 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.671459 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.702118 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.702076 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfdd13-openshift-default-dockercfg-jf87f\"" Nov 01 10:02:18.772895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.772853 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.772895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.772896 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773142 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.772913 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773142 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773046 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773142 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773095 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773142 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773122 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773325 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773175 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773325 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773252 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7d6w\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773424 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773324 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773424 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773348 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773534 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773430 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773534 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773508 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.773687 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.773551 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.774112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.774040 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.775854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.775826 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.776004 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.775986 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.832089 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.832004 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.832089 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.832082 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7d6w\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w\") pod \"e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:18.877490 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:18.877453 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:19.033840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:19.033790 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" podUID="a81fbead-001d-408c-946b-0bff9281b071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:02:19.301317 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:19.301281 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h"] Nov 01 10:02:19.304972 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:02:19.304932 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18fd384_9f5b_4737_9b3d_1e9775fc1ab8.slice/crio-50a9ff4f8d4ef34aa2f0ad89ace8933124e8fc719a54faf1f70018baf50a765c WatchSource:0}: Error finding container 50a9ff4f8d4ef34aa2f0ad89ace8933124e8fc719a54faf1f70018baf50a765c: Status 404 returned error can't find the container with id 50a9ff4f8d4ef34aa2f0ad89ace8933124e8fc719a54faf1f70018baf50a765c Nov 01 10:02:19.307152 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:19.307123 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:02:19.307236 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:19.307194 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:02:19.307293 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:19.307235 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:02:20.118429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:20.118387 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" event={"ID":"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8","Type":"ContainerStarted","Data":"41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a"} Nov 01 10:02:20.118895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:20.118452 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" event={"ID":"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8","Type":"ContainerStarted","Data":"50a9ff4f8d4ef34aa2f0ad89ace8933124e8fc719a54faf1f70018baf50a765c"} Nov 01 10:02:20.234197 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:20.234101 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podStartSLOduration=2.234084783 podStartE2EDuration="2.234084783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:02:20.233150856 +0000 UTC m=+5363.321487537" watchObservedRunningTime="2026-11-01 10:02:20.234084783 +0000 UTC m=+5363.322421454" Nov 01 10:02:20.878552 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:20.878517 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:20.879908 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:20.879880 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 01 10:02:20.880051 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:20.879929 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 01 10:02:21.878099 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:21.878065 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 01 10:02:21.878535 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:21.878119 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 01 10:02:22.877979 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:22.877940 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 01 10:02:22.878172 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:22.877996 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 01 10:02:23.878179 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:23.878141 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 01 10:02:23.878610 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:23.878206 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 01 10:02:24.157547 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.157506 5710 generic.go:356] "Generic (PLEG): container finished" podID="a81fbead-001d-408c-946b-0bff9281b071" containerID="4ede7eaa58dc1569cb14752cfdd04191193fb75eac6a0b0286a77084580fc7b7" exitCode=0 Nov 01 10:02:24.157766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.157672 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" event={"ID":"a81fbead-001d-408c-946b-0bff9281b071","Type":"ContainerDied","Data":"4ede7eaa58dc1569cb14752cfdd04191193fb75eac6a0b0286a77084580fc7b7"} Nov 01 10:02:24.236378 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.236350 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 10:02:24.250518 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.250479 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d\" (UniqueName: \"kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d\") pod \"a81fbead-001d-408c-946b-0bff9281b071\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " Nov 01 10:02:24.250758 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.250547 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume\") pod \"a81fbead-001d-408c-946b-0bff9281b071\" (UID: \"a81fbead-001d-408c-946b-0bff9281b071\") " Nov 01 10:02:24.251022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.250998 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume" pod "a81fbead-001d-408c-946b-0bff9281b071" (UID: "a81fbead-001d-408c-946b-0bff9281b071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:02:24.253062 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.253036 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d" pod "a81fbead-001d-408c-946b-0bff9281b071" (UID: "a81fbead-001d-408c-946b-0bff9281b071"). InnerVolumeSpecName "kube-api-access-4qt7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:24.351559 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.351523 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qt7d\" (UniqueName: \"kubernetes.io/projected/a81fbead-001d-408c-946b-0bff9281b071-kube-api-access-4qt7d\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:24.351559 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.351556 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81fbead-001d-408c-946b-0bff9281b071-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:24.878002 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.877967 5710 patch_prober.go:29] interesting pod/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 01 10:02:24.878176 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:24.878033 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 01 10:02:25.163605 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.163499 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" event={"ID":"a81fbead-001d-408c-946b-0bff9281b071","Type":"ContainerDied","Data":"6f47eb1256567ec4c642013ef5068219ad25ffc1432ee05a359b1478a8c141b5"} Nov 01 10:02:25.163605 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.163570 5710 scope.go:122] "RemoveContainer" containerID="4ede7eaa58dc1569cb14752cfdd04191193fb75eac6a0b0286a77084580fc7b7" Nov 01 10:02:25.163605 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.163508 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv" Nov 01 10:02:25.317005 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.316968 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv"] Nov 01 10:02:25.322698 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.322668 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfdd13-6bc6bcb88c-r94rv"] Nov 01 10:02:25.882381 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.882350 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:25.882679 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.882657 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:25.883507 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:25.883485 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:27.056831 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:27.056798 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81fbead-001d-408c-946b-0bff9281b071" path="/var/lib/kubelet/pods/a81fbead-001d-408c-946b-0bff9281b071/volumes" Nov 01 10:02:31.917008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:31.916973 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h"] Nov 01 10:02:31.917898 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:31.917842 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" containerID="cri-o://41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a" gracePeriod=30 Nov 01 10:02:33.037249 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:33.037207 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c65545f6c-spmfl_1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa/authorino/0.log" Nov 01 10:02:35.516570 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:35.516541 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2rfzc_308c91a4-a197-4420-8d55-7d7e9fe1e44d/limitador/0.log" Nov 01 10:02:37.186261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.186182 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:37.226049 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.225993 5710 generic.go:356] "Generic (PLEG): container finished" podID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerID="41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a" exitCode=0 Nov 01 10:02:37.226319 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.226068 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" Nov 01 10:02:37.226319 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.226071 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" event={"ID":"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8","Type":"ContainerDied","Data":"41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a"} Nov 01 10:02:37.226319 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.226216 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h" event={"ID":"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8","Type":"ContainerDied","Data":"50a9ff4f8d4ef34aa2f0ad89ace8933124e8fc719a54faf1f70018baf50a765c"} Nov 01 10:02:37.226319 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.226252 5710 scope.go:122] "RemoveContainer" containerID="41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a" Nov 01 10:02:37.238103 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.238078 5710 scope.go:122] "RemoveContainer" containerID="41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a" Nov 01 10:02:37.238522 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:37.238497 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a\": container with ID starting with 41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a not found: ID does not exist" containerID="41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a" Nov 01 10:02:37.238665 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.238532 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a"} err="failed to get container status \"41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a\": rpc error: code = NotFound desc = could not find container \"41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a\": container with ID starting with 41e32b7d84ea77c296f30cc2f340f4feb7a2f432e6a245de467684d16a62ba7a not found: ID does not exist" Nov 01 10:02:37.286628 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286591 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.286861 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286644 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.286861 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286678 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.286861 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286775 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.286861 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286821 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.287150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286901 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.287150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286906 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:37.287150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286952 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.287150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.286992 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.287150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287047 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token\") pod \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\" (UID: \"c18fd384-9f5b-4737-9b3d-1e9775fc1ab8\") " Nov 01 10:02:37.287477 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287280 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:37.287477 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287329 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:37.287657 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287631 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.287657 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287659 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.287888 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287673 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.287888 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.287810 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:02:37.288316 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.288100 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:37.289969 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.289942 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:37.290094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.290057 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 10:02:37.290267 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.290248 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:02:37.290363 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.290323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w" pod "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" (UID: "c18fd384-9f5b-4737-9b3d-1e9775fc1ab8"). InnerVolumeSpecName "kube-api-access-r7d6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:02:37.388735 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.388684 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.388735 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.388734 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.388974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.388749 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7d6w\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-kube-api-access-r7d6w\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.388974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.388764 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.388974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.388776 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.388974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.388787 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:02:37.654472 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.654430 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h"] Nov 01 10:02:37.657963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:37.657933 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfdd13-openshift-default-57fb7bb57f-44s2h"] Nov 01 10:02:38.120402 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:38.120365 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7657794547-t5tw5_ab7ffc9a-2189-49be-adcb-2ff757eb57d5/manager/0.log" Nov 01 10:02:39.057952 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:39.057914 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" path="/var/lib/kubelet/pods/c18fd384-9f5b-4737-9b3d-1e9775fc1ab8/volumes" Nov 01 10:02:39.807678 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:39.807643 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5w7w2_85247c14-ee89-4497-9d63-7518733b0a98/manager/0.log" Nov 01 10:02:45.478019 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:45.477978 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:02:54.426119 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426078 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd"] Nov 01 10:02:54.426551 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426527 5710 cpu_manager.go:414] "containerMap: removing container" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" Nov 01 10:02:54.426551 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426538 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" Nov 01 10:02:54.426633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426555 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a81fbead-001d-408c-946b-0bff9281b071" containerName="payload-pre-processing" Nov 01 10:02:54.426633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426563 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81fbead-001d-408c-946b-0bff9281b071" containerName="payload-pre-processing" Nov 01 10:02:54.426795 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426660 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18fd384-9f5b-4737-9b3d-1e9775fc1ab8" containerName="istio-proxy" Nov 01 10:02:54.426795 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.426669 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81fbead-001d-408c-946b-0bff9281b071" containerName="payload-pre-processing" Nov 01 10:02:54.429894 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.429873 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.486039 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.486002 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7e1b7946-serving-cert\"" Nov 01 10:02:54.532565 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.532523 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7zt8\"" Nov 01 10:02:54.532565 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.532529 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 10:02:54.532565 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.532538 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:02:54.532847 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.532587 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:02:54.534156 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.534136 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd"] Nov 01 10:02:54.560436 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.560408 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.560617 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.560452 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.560617 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.560521 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zkq\" (UniqueName: \"kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.560748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.560698 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.670942 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.667101 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zkq\" (UniqueName: \"kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.670942 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.667778 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.670942 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.667948 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.670942 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:54.668212 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-serving-cert: secret "maas-api-e2e-ait-7e1b7946-serving-cert" not found Nov 01 10:02:54.670942 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:02:54.668290 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls podName:a3f36098-e140-49e8-947b-6190e9650885 nodeName:}" failed. No retries permitted until 2026-11-01 10:02:55.168268854 +0000 UTC m=+5398.256605519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls") pod "maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" (UID: "a3f36098-e140-49e8-947b-6190e9650885") : secret "maas-api-e2e-ait-7e1b7946-serving-cert" not found Nov 01 10:02:54.670942 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.669020 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.671572 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.671443 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.673350 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.673322 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:54.728107 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:54.728024 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zkq\" (UniqueName: \"kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:55.174683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:55.174637 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:55.177520 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:55.177494 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\") pod \"maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:55.342497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:55.342442 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:55.720669 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:55.720628 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd"] Nov 01 10:02:55.724745 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:02:55.724694 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f36098_e140_49e8_947b_6190e9650885.slice/crio-d05e075504df4d9cb92b052d43b9a6cccfb35837aeb53e64022758df31c02db8 WatchSource:0}: Error finding container d05e075504df4d9cb92b052d43b9a6cccfb35837aeb53e64022758df31c02db8: Status 404 returned error can't find the container with id d05e075504df4d9cb92b052d43b9a6cccfb35837aeb53e64022758df31c02db8 Nov 01 10:02:55.726244 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:55.726222 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 10:02:56.322465 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:56.322427 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" event={"ID":"a3f36098-e140-49e8-947b-6190e9650885","Type":"ContainerStarted","Data":"d05e075504df4d9cb92b052d43b9a6cccfb35837aeb53e64022758df31c02db8"} Nov 01 10:02:57.329136 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:57.329095 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" event={"ID":"a3f36098-e140-49e8-947b-6190e9650885","Type":"ContainerStarted","Data":"1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234"} Nov 01 10:02:57.329594 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:57.329206 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:02:57.444498 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:02:57.444442 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" podStartSLOduration=2.427874425 podStartE2EDuration="3.444425019s" totalImagesPullingTime="1.016550594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:02:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:02:57.442445353 +0000 UTC m=+5400.530782025" watchObservedRunningTime="2026-11-01 10:02:57.444425019 +0000 UTC m=+5400.532761690" Nov 01 10:03:01.404148 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:01.404106 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:03:01.416049 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:01.416020 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:03:02.287808 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:02.287768 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:03:03.340399 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:03.340367 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:03:15.477473 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:03:15.477429 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:03:43.469839 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.469798 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd"] Nov 01 10:03:43.470438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.470041 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" podUID="a3f36098-e140-49e8-947b-6190e9650885" containerName="maas-api" containerID="cri-o://1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234" gracePeriod=30 Nov 01 10:03:43.734792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.734706 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:03:43.866680 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.866642 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\") pod \"a3f36098-e140-49e8-947b-6190e9650885\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " Nov 01 10:03:43.866909 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.866696 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq\" (UniqueName: \"kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq\") pod \"a3f36098-e140-49e8-947b-6190e9650885\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " Nov 01 10:03:43.866909 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.866865 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca\") pod \"a3f36098-e140-49e8-947b-6190e9650885\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " Nov 01 10:03:43.867041 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.866929 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs\") pod \"a3f36098-e140-49e8-947b-6190e9650885\" (UID: \"a3f36098-e140-49e8-947b-6190e9650885\") " Nov 01 10:03:43.867261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.867232 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca" pod "a3f36098-e140-49e8-947b-6190e9650885" (UID: "a3f36098-e140-49e8-947b-6190e9650885"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:03:43.869254 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.869218 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls" pod "a3f36098-e140-49e8-947b-6190e9650885" (UID: "a3f36098-e140-49e8-947b-6190e9650885"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:03:43.869254 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.869233 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq" pod "a3f36098-e140-49e8-947b-6190e9650885" (UID: "a3f36098-e140-49e8-947b-6190e9650885"). InnerVolumeSpecName "kube-api-access-g4zkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:03:43.869555 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.869531 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs" pod "a3f36098-e140-49e8-947b-6190e9650885" (UID: "a3f36098-e140-49e8-947b-6190e9650885"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:03:43.968340 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.968309 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:03:43.968340 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.968337 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4zkq\" (UniqueName: \"kubernetes.io/projected/a3f36098-e140-49e8-947b-6190e9650885-kube-api-access-g4zkq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:03:43.968559 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.968349 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f36098-e140-49e8-947b-6190e9650885-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:03:43.968559 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:43.968361 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f36098-e140-49e8-947b-6190e9650885-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:03:44.559584 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.559544 5710 generic.go:356] "Generic (PLEG): container finished" podID="a3f36098-e140-49e8-947b-6190e9650885" containerID="1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234" exitCode=0 Nov 01 10:03:44.560154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.559594 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" Nov 01 10:03:44.560154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.559610 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" event={"ID":"a3f36098-e140-49e8-947b-6190e9650885","Type":"ContainerDied","Data":"1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234"} Nov 01 10:03:44.560154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.559666 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd" event={"ID":"a3f36098-e140-49e8-947b-6190e9650885","Type":"ContainerDied","Data":"d05e075504df4d9cb92b052d43b9a6cccfb35837aeb53e64022758df31c02db8"} Nov 01 10:03:44.560154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.559700 5710 scope.go:122] "RemoveContainer" containerID="1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234" Nov 01 10:03:44.569935 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.569913 5710 scope.go:122] "RemoveContainer" containerID="1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234" Nov 01 10:03:44.570212 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:03:44.570194 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234\": container with ID starting with 1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234 not found: ID does not exist" containerID="1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234" Nov 01 10:03:44.570283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.570226 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234"} err="failed to get container status \"1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234\": rpc error: code = NotFound desc = could not find container \"1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234\": container with ID starting with 1a030f163ffa24ae091f70890a4067506a17ce1f22e48db910f5ab6cd1b0a234 not found: ID does not exist" Nov 01 10:03:44.713825 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.713790 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd"] Nov 01 10:03:44.760264 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:44.760228 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e1b7946-7f9c5c64d8-56gnd"] Nov 01 10:03:45.056946 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:45.056902 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f36098-e140-49e8-947b-6190e9650885" path="/var/lib/kubelet/pods/a3f36098-e140-49e8-947b-6190e9650885/volumes" Nov 01 10:03:45.477145 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:03:45.477111 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:03:47.865514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:03:47.865476 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:04:15.476871 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:04:15.476827 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:04:20.866602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.866562 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd"] Nov 01 10:04:20.867087 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.867069 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a3f36098-e140-49e8-947b-6190e9650885" containerName="maas-api" Nov 01 10:04:20.867135 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.867089 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f36098-e140-49e8-947b-6190e9650885" containerName="maas-api" Nov 01 10:04:20.867187 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.867177 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f36098-e140-49e8-947b-6190e9650885" containerName="maas-api" Nov 01 10:04:20.870441 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.870424 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948035 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.947992 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948035 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948033 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948062 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98262e95-96f5-4250-ac07-edfdb9334102-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948177 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948197 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5dd\" (UniqueName: \"kubernetes.io/projected/98262e95-96f5-4250-ac07-edfdb9334102-kube-api-access-4t5dd\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948216 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948443 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948289 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98262e95-96f5-4250-ac07-edfdb9334102-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948443 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948343 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.948443 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.948401 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98262e95-96f5-4250-ac07-edfdb9334102-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:20.967393 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:20.967362 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd"] Nov 01 10:04:21.049002 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.048967 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98262e95-96f5-4250-ac07-edfdb9334102-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049209 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049021 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049209 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049050 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98262e95-96f5-4250-ac07-edfdb9334102-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049297 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049218 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049297 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049265 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049422 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049296 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98262e95-96f5-4250-ac07-edfdb9334102-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049422 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049333 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049422 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049406 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5dd\" (UniqueName: \"kubernetes.io/projected/98262e95-96f5-4250-ac07-edfdb9334102-kube-api-access-4t5dd\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049449 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049669 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049620 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049754 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049697 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.049974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.049924 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.050169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.050145 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.050254 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.050200 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98262e95-96f5-4250-ac07-edfdb9334102-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.051766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.051743 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98262e95-96f5-4250-ac07-edfdb9334102-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.051766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.051754 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98262e95-96f5-4250-ac07-edfdb9334102-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.107298 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.107261 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98262e95-96f5-4250-ac07-edfdb9334102-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.107443 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.107383 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5dd\" (UniqueName: \"kubernetes.io/projected/98262e95-96f5-4250-ac07-edfdb9334102-kube-api-access-4t5dd\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vmqkd\" (UID: \"98262e95-96f5-4250-ac07-edfdb9334102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.184082 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.183992 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:21.616347 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.616319 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd"] Nov 01 10:04:21.619157 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:04:21.619124 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98262e95_96f5_4250_ac07_edfdb9334102.slice/crio-3d23974e6d44fd0e7117c98dca4c2c17005bb864abcf03a408824b70ecb650b5 WatchSource:0}: Error finding container 3d23974e6d44fd0e7117c98dca4c2c17005bb864abcf03a408824b70ecb650b5: Status 404 returned error can't find the container with id 3d23974e6d44fd0e7117c98dca4c2c17005bb864abcf03a408824b70ecb650b5 Nov 01 10:04:21.621911 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.621877 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:04:21.622015 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.621951 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:04:21.622015 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.621983 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:04:21.749592 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.749546 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" event={"ID":"98262e95-96f5-4250-ac07-edfdb9334102","Type":"ContainerStarted","Data":"9ff052dd9e5cbcd5abd2eff4a2b3166583cea6807610e3e0f7902f9f83d48494"} Nov 01 10:04:21.749856 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.749610 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" event={"ID":"98262e95-96f5-4250-ac07-edfdb9334102","Type":"ContainerStarted","Data":"3d23974e6d44fd0e7117c98dca4c2c17005bb864abcf03a408824b70ecb650b5"} Nov 01 10:04:21.865581 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:21.865520 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podStartSLOduration=1.8655025269999999 podStartE2EDuration="1.865502527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:04:21.863621447 +0000 UTC m=+5484.951958118" watchObservedRunningTime="2026-11-01 10:04:21.865502527 +0000 UTC m=+5484.953839258" Nov 01 10:04:22.201748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:22.196806 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:22.201748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:22.197016 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:22.201748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:22.197109 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:23.185251 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:23.185208 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:23.185602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:23.185567 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:24.185742 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:24.185092 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:24.185742 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:24.185153 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:25.196193 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:25.196149 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:25.196873 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:25.196838 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:26.196253 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:26.196207 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:26.196956 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:26.196916 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:27.193749 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:27.188149 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:27.193749 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:27.188221 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:28.185248 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:28.185197 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:28.185692 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:28.185281 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:29.184774 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:29.184701 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 10:04:29.184992 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:29.184797 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 10:04:31.185375 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:31.185324 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:04:31.185922 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:31.185399 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:04:32.185934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:32.185869 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:04:32.186411 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:32.185940 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:04:33.227898 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:33.227855 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:04:33.228469 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:33.227912 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:04:33.854287 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:33.854257 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:33.854564 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:33.854542 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:33.855299 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:33.855276 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" Nov 01 10:04:45.477798 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:04:45.477758 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:04:59.547629 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:04:59.547592 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:05:06.256170 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.256131 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl"] Nov 01 10:05:06.260120 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.260101 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.313099 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.313062 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:05:06.313357 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.313109 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:05:06.359905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.359860 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:05:06.359905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.359885 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:05:06.361527 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.361502 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl"] Nov 01 10:05:06.450852 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.450805 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.450852 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.450861 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xph\" (UniqueName: \"kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.451099 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.450949 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.552214 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.552115 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.552214 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.552164 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xph\" (UniqueName: \"kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.552214 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.552187 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.552838 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.552812 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.554952 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.554923 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.611248 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.611198 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xph\" (UniqueName: \"kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph\") pod \"maas-api-revoke-sub-f904bff9-wcgzl\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:06.872945 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:06.872899 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:07.247013 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:07.246933 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl"] Nov 01 10:05:07.250209 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:05:07.250162 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f27c92_a19c_4af4_bea4_ae948efae41c.slice/crio-827600b8f7c6125b9347d61f7d465b11c7a954eba0fab847f6c6b6fd3234c0d0 WatchSource:0}: Error finding container 827600b8f7c6125b9347d61f7d465b11c7a954eba0fab847f6c6b6fd3234c0d0: Status 404 returned error can't find the container with id 827600b8f7c6125b9347d61f7d465b11c7a954eba0fab847f6c6b6fd3234c0d0 Nov 01 10:05:07.981523 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:07.981480 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" event={"ID":"e5f27c92-a19c-4af4-bea4-ae948efae41c","Type":"ContainerStarted","Data":"672f56b0e2904980b171bcd331ff6dec1be25c3ce83854a8885ed844c8ec4551"} Nov 01 10:05:07.982005 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:07.981547 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" event={"ID":"e5f27c92-a19c-4af4-bea4-ae948efae41c","Type":"ContainerStarted","Data":"827600b8f7c6125b9347d61f7d465b11c7a954eba0fab847f6c6b6fd3234c0d0"} Nov 01 10:05:08.092591 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:08.092532 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" podStartSLOduration=2.092512602 podStartE2EDuration="2.092512602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:05:08.090921876 +0000 UTC m=+5531.179258565" watchObservedRunningTime="2026-11-01 10:05:08.092512602 +0000 UTC m=+5531.180849274" Nov 01 10:05:08.988306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:08.988270 5710 generic.go:356] "Generic (PLEG): container finished" podID="e5f27c92-a19c-4af4-bea4-ae948efae41c" containerID="672f56b0e2904980b171bcd331ff6dec1be25c3ce83854a8885ed844c8ec4551" exitCode=0 Nov 01 10:05:08.988736 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:08.988353 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" event={"ID":"e5f27c92-a19c-4af4-bea4-ae948efae41c","Type":"ContainerDied","Data":"672f56b0e2904980b171bcd331ff6dec1be25c3ce83854a8885ed844c8ec4551"} Nov 01 10:05:10.136096 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.136068 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:10.188905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.188873 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph\" (UniqueName: \"kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph\") pod \"e5f27c92-a19c-4af4-bea4-ae948efae41c\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " Nov 01 10:05:10.189088 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.188923 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca\") pod \"e5f27c92-a19c-4af4-bea4-ae948efae41c\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " Nov 01 10:05:10.189088 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.189044 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca\") pod \"e5f27c92-a19c-4af4-bea4-ae948efae41c\" (UID: \"e5f27c92-a19c-4af4-bea4-ae948efae41c\") " Nov 01 10:05:10.189318 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.189292 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca" pod "e5f27c92-a19c-4af4-bea4-ae948efae41c" (UID: "e5f27c92-a19c-4af4-bea4-ae948efae41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:05:10.189495 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.189435 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:05:10.191313 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.191283 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca" pod "e5f27c92-a19c-4af4-bea4-ae948efae41c" (UID: "e5f27c92-a19c-4af4-bea4-ae948efae41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:05:10.191419 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.191359 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph" pod "e5f27c92-a19c-4af4-bea4-ae948efae41c" (UID: "e5f27c92-a19c-4af4-bea4-ae948efae41c"). InnerVolumeSpecName "kube-api-access-p6xph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:05:10.290582 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.290486 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xph\" (UniqueName: \"kubernetes.io/projected/e5f27c92-a19c-4af4-bea4-ae948efae41c-kube-api-access-p6xph\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:05:10.290582 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.290523 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f27c92-a19c-4af4-bea4-ae948efae41c-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:05:10.999730 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.999684 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" Nov 01 10:05:10.999943 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.999709 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl" event={"ID":"e5f27c92-a19c-4af4-bea4-ae948efae41c","Type":"ContainerDied","Data":"827600b8f7c6125b9347d61f7d465b11c7a954eba0fab847f6c6b6fd3234c0d0"} Nov 01 10:05:10.999943 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:10.999819 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827600b8f7c6125b9347d61f7d465b11c7a954eba0fab847f6c6b6fd3234c0d0" Nov 01 10:05:15.476852 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:05:15.476810 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:05:18.898163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:18.898122 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:05:21.325031 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.324987 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg"] Nov 01 10:05:21.325470 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.325457 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e5f27c92-a19c-4af4-bea4-ae948efae41c" containerName="revoke-keys" Nov 01 10:05:21.325516 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.325474 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f27c92-a19c-4af4-bea4-ae948efae41c" containerName="revoke-keys" Nov 01 10:05:21.325569 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.325558 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f27c92-a19c-4af4-bea4-ae948efae41c" containerName="revoke-keys" Nov 01 10:05:21.328944 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.328921 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.404650 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.404600 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kx6\" (UniqueName: \"kubernetes.io/projected/2aed84f7-2b75-4239-b5f3-cbc504023f8e-kube-api-access-h9kx6\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.404650 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.404646 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.404952 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.404753 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.404952 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.404914 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.405052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.404963 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.405052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.405036 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.405149 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.405124 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.405205 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.405193 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.405240 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.405214 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.427246 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.427194 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg"] Nov 01 10:05:21.506771 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.506697 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kx6\" (UniqueName: \"kubernetes.io/projected/2aed84f7-2b75-4239-b5f3-cbc504023f8e-kube-api-access-h9kx6\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.506771 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.506774 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.506816 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.506882 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.506910 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.506942 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507016 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507354 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507066 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507354 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507094 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507472 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507405 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507556 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507529 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507623 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507554 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507805 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507778 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.507892 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.507818 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.509591 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.509570 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.509742 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.509708 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.565735 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.565694 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aed84f7-2b75-4239-b5f3-cbc504023f8e-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.565937 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.565869 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kx6\" (UniqueName: \"kubernetes.io/projected/2aed84f7-2b75-4239-b5f3-cbc504023f8e-kube-api-access-h9kx6\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hkrgg\" (UID: \"2aed84f7-2b75-4239-b5f3-cbc504023f8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:21.643399 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:21.643357 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:22.069368 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:22.069267 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg"] Nov 01 10:05:22.080980 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:05:22.080936 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aed84f7_2b75_4239_b5f3_cbc504023f8e.slice/crio-2542a761fc89e69cee63c4691e3f4a4f4316b5ebb11340c2208179e085e9b07c WatchSource:0}: Error finding container 2542a761fc89e69cee63c4691e3f4a4f4316b5ebb11340c2208179e085e9b07c: Status 404 returned error can't find the container with id 2542a761fc89e69cee63c4691e3f4a4f4316b5ebb11340c2208179e085e9b07c Nov 01 10:05:22.083398 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:22.083363 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:05:22.083486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:22.083435 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:05:22.083486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:22.083465 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:05:23.060501 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:23.060460 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" event={"ID":"2aed84f7-2b75-4239-b5f3-cbc504023f8e","Type":"ContainerStarted","Data":"3d5582ee647df68aee7d3e2564c895ba49ae839014d503d1d2100f9ab1fc48ba"} Nov 01 10:05:23.061018 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:23.060523 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" event={"ID":"2aed84f7-2b75-4239-b5f3-cbc504023f8e","Type":"ContainerStarted","Data":"2542a761fc89e69cee63c4691e3f4a4f4316b5ebb11340c2208179e085e9b07c"} Nov 01 10:05:23.221916 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:23.221817 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podStartSLOduration=2.221765925 podStartE2EDuration="2.221765925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:05:23.220666207 +0000 UTC m=+5546.309002893" watchObservedRunningTime="2026-11-01 10:05:23.221765925 +0000 UTC m=+5546.310102598" Nov 01 10:05:23.644231 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:23.644187 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:23.645966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:23.645929 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:23.646146 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:23.646002 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:24.644669 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:24.644621 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:24.645133 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:24.644705 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:25.644137 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:25.644095 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:25.644469 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:25.644427 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:26.646224 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:26.646183 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:26.646873 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:26.646833 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:27.644737 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:27.644682 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:27.644954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:27.644786 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:28.563571 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:28.563531 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:05:28.643703 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:28.643661 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:28.643940 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:28.643753 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:29.644085 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:29.644041 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 10:05:29.644555 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:29.644117 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 10:05:31.645078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:31.645032 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:31.645557 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:31.645112 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:32.645832 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:32.645790 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:32.646290 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:32.645855 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:33.688935 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:33.688892 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:33.689541 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:33.689500 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:34.690689 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:34.690628 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:34.691202 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:34.690734 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:35.703747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:35.703014 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 10:05:35.703747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:35.703294 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 10:05:36.531747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:36.527462 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:05:36.643940 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:36.643878 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:36.753420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:36.752840 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:36.753420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:36.752934 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:37.795476 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:37.795414 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:37.796392 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:37.796354 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:38.309499 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:38.309466 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:38.432817 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:38.432776 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" Nov 01 10:05:39.779852 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.779819 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n"] Nov 01 10:05:39.783777 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.783757 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.835550 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.835514 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:05:39.871140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.871086 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.871355 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.871292 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmv7x\" (UniqueName: \"kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.871404 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.871350 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.882018 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.881982 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:05:39.882222 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.882029 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:05:39.882222 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.882054 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:05:39.883634 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.883612 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n"] Nov 01 10:05:39.972797 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.972756 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.973052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.972842 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmv7x\" (UniqueName: \"kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.973052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.972878 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.973538 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.973518 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:39.975452 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:39.975427 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:40.031810 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:40.031708 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmv7x\" (UniqueName: \"kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x\") pod \"maas-api-revoke-sub-f03629db-6999n\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:40.097081 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:40.097039 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:40.468301 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:40.468268 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n"] Nov 01 10:05:40.471749 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:05:40.471696 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800feaee_0811_482b_a561_8eb3b53f4266.slice/crio-c0d04591cdbe65801da3d333988274231d0e56516c4dddf9898f175d1f5a9cd4 WatchSource:0}: Error finding container c0d04591cdbe65801da3d333988274231d0e56516c4dddf9898f175d1f5a9cd4: Status 404 returned error can't find the container with id c0d04591cdbe65801da3d333988274231d0e56516c4dddf9898f175d1f5a9cd4 Nov 01 10:05:41.186146 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:41.186110 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" event={"ID":"800feaee-0811-482b-a561-8eb3b53f4266","Type":"ContainerStarted","Data":"dc02e1547de484737ce4a88ce01047b16a5a2ee60e30985465d0c45384ca344e"} Nov 01 10:05:41.186631 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:41.186173 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" event={"ID":"800feaee-0811-482b-a561-8eb3b53f4266","Type":"ContainerStarted","Data":"c0d04591cdbe65801da3d333988274231d0e56516c4dddf9898f175d1f5a9cd4"} Nov 01 10:05:41.295743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:41.295657 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" podStartSLOduration=2.295638716 podStartE2EDuration="2.295638716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:05:41.294868625 +0000 UTC m=+5564.383205296" watchObservedRunningTime="2026-11-01 10:05:41.295638716 +0000 UTC m=+5564.383975388" Nov 01 10:05:42.192584 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:42.192549 5710 generic.go:356] "Generic (PLEG): container finished" podID="800feaee-0811-482b-a561-8eb3b53f4266" containerID="dc02e1547de484737ce4a88ce01047b16a5a2ee60e30985465d0c45384ca344e" exitCode=0 Nov 01 10:05:42.193103 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:42.192613 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" event={"ID":"800feaee-0811-482b-a561-8eb3b53f4266","Type":"ContainerDied","Data":"dc02e1547de484737ce4a88ce01047b16a5a2ee60e30985465d0c45384ca344e"} Nov 01 10:05:43.349871 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.349843 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:43.508110 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.508020 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca\") pod \"800feaee-0811-482b-a561-8eb3b53f4266\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " Nov 01 10:05:43.508277 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.508167 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x\" (UniqueName: \"kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x\") pod \"800feaee-0811-482b-a561-8eb3b53f4266\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " Nov 01 10:05:43.508277 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.508193 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca\") pod \"800feaee-0811-482b-a561-8eb3b53f4266\" (UID: \"800feaee-0811-482b-a561-8eb3b53f4266\") " Nov 01 10:05:43.508492 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.508452 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca" pod "800feaee-0811-482b-a561-8eb3b53f4266" (UID: "800feaee-0811-482b-a561-8eb3b53f4266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:05:43.510599 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.510566 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca" pod "800feaee-0811-482b-a561-8eb3b53f4266" (UID: "800feaee-0811-482b-a561-8eb3b53f4266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:05:43.510599 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.510573 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x" pod "800feaee-0811-482b-a561-8eb3b53f4266" (UID: "800feaee-0811-482b-a561-8eb3b53f4266"). InnerVolumeSpecName "kube-api-access-cmv7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:05:43.609965 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.609918 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmv7x\" (UniqueName: \"kubernetes.io/projected/800feaee-0811-482b-a561-8eb3b53f4266-kube-api-access-cmv7x\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:05:43.609965 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.609954 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:05:43.609965 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:43.609964 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800feaee-0811-482b-a561-8eb3b53f4266-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:05:44.204864 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:44.204829 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" Nov 01 10:05:44.204864 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:44.204856 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n" event={"ID":"800feaee-0811-482b-a561-8eb3b53f4266","Type":"ContainerDied","Data":"c0d04591cdbe65801da3d333988274231d0e56516c4dddf9898f175d1f5a9cd4"} Nov 01 10:05:44.205101 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:44.204906 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0d04591cdbe65801da3d333988274231d0e56516c4dddf9898f175d1f5a9cd4" Nov 01 10:05:44.879439 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:44.879394 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:05:45.478913 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:05:45.478866 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:05:57.669872 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:57.669828 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:05:57.670281 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:57.669895 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:05:59.498610 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.498568 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht"] Nov 01 10:05:59.499260 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.499236 5710 cpu_manager.go:414] "containerMap: removing container" podUID="800feaee-0811-482b-a561-8eb3b53f4266" containerName="revoke-keys" Nov 01 10:05:59.499369 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.499262 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800feaee-0811-482b-a561-8eb3b53f4266" containerName="revoke-keys" Nov 01 10:05:59.499429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.499416 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="800feaee-0811-482b-a561-8eb3b53f4266" containerName="revoke-keys" Nov 01 10:05:59.503150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.503120 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.600640 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.600595 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:05:59.600640 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.600602 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:05:59.600640 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.600602 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:05:59.600967 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.600653 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:05:59.602321 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.602298 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht"] Nov 01 10:05:59.618131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.618089 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkl57\" (UniqueName: \"kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.618250 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.618215 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.618374 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.618358 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.719735 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.719669 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.720009 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.719889 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.720009 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.719931 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl57\" (UniqueName: \"kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.720409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.720382 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.722780 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.722746 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.778617 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.778533 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl57\" (UniqueName: \"kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57\") pod \"maas-api-revoke-sub-0165d7d2-lplht\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:05:59.817974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:05:59.817930 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:06:00.193109 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:00.193069 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht"] Nov 01 10:06:00.196808 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:06:00.196766 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7710ad5f_b40e_436c_a12f_acc51da9319e.slice/crio-413ea37252863666dddc82281101d675d4606fd7b7d56c4330893727604a1a4d WatchSource:0}: Error finding container 413ea37252863666dddc82281101d675d4606fd7b7d56c4330893727604a1a4d: Status 404 returned error can't find the container with id 413ea37252863666dddc82281101d675d4606fd7b7d56c4330893727604a1a4d Nov 01 10:06:00.290730 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:00.290682 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" event={"ID":"7710ad5f-b40e-436c-a12f-acc51da9319e","Type":"ContainerStarted","Data":"413ea37252863666dddc82281101d675d4606fd7b7d56c4330893727604a1a4d"} Nov 01 10:06:01.297579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:01.297540 5710 generic.go:356] "Generic (PLEG): container finished" podID="7710ad5f-b40e-436c-a12f-acc51da9319e" containerID="f2ae477500663c9a077e08991db58466a3b8d32cb201f3ae64600bac803a6427" exitCode=0 Nov 01 10:06:01.298056 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:01.297630 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" event={"ID":"7710ad5f-b40e-436c-a12f-acc51da9319e","Type":"ContainerDied","Data":"f2ae477500663c9a077e08991db58466a3b8d32cb201f3ae64600bac803a6427"} Nov 01 10:06:02.445825 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.445796 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:06:02.546734 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.546683 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57\" (UniqueName: \"kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57\") pod \"7710ad5f-b40e-436c-a12f-acc51da9319e\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " Nov 01 10:06:02.546941 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.546759 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca\") pod \"7710ad5f-b40e-436c-a12f-acc51da9319e\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " Nov 01 10:06:02.546941 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.546857 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca\") pod \"7710ad5f-b40e-436c-a12f-acc51da9319e\" (UID: \"7710ad5f-b40e-436c-a12f-acc51da9319e\") " Nov 01 10:06:02.547256 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.547231 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca" pod "7710ad5f-b40e-436c-a12f-acc51da9319e" (UID: "7710ad5f-b40e-436c-a12f-acc51da9319e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:06:02.549116 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.549093 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca" pod "7710ad5f-b40e-436c-a12f-acc51da9319e" (UID: "7710ad5f-b40e-436c-a12f-acc51da9319e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:06:02.549216 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.549129 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57" pod "7710ad5f-b40e-436c-a12f-acc51da9319e" (UID: "7710ad5f-b40e-436c-a12f-acc51da9319e"). InnerVolumeSpecName "kube-api-access-gkl57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:06:02.648112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.648067 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkl57\" (UniqueName: \"kubernetes.io/projected/7710ad5f-b40e-436c-a12f-acc51da9319e-kube-api-access-gkl57\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:06:02.648112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.648104 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:06:02.648112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:02.648119 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7710ad5f-b40e-436c-a12f-acc51da9319e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:06:03.308974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:03.308938 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" Nov 01 10:06:03.308974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:03.308964 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht" event={"ID":"7710ad5f-b40e-436c-a12f-acc51da9319e","Type":"ContainerDied","Data":"413ea37252863666dddc82281101d675d4606fd7b7d56c4330893727604a1a4d"} Nov 01 10:06:03.309215 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:03.309022 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="413ea37252863666dddc82281101d675d4606fd7b7d56c4330893727604a1a4d" Nov 01 10:06:11.199765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.199700 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb"] Nov 01 10:06:11.200304 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.200152 5710 cpu_manager.go:414] "containerMap: removing container" podUID="7710ad5f-b40e-436c-a12f-acc51da9319e" containerName="revoke-keys" Nov 01 10:06:11.200304 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.200163 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7710ad5f-b40e-436c-a12f-acc51da9319e" containerName="revoke-keys" Nov 01 10:06:11.200304 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.200247 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="7710ad5f-b40e-436c-a12f-acc51da9319e" containerName="revoke-keys" Nov 01 10:06:11.203356 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.203337 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.240224 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.240171 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpn4t\" (UniqueName: \"kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.240428 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.240341 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.240476 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.240445 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.252772 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.252746 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:06:11.255265 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.255242 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:06:11.301600 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.301565 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:06:11.301822 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.301660 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:06:11.303317 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.303300 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb"] Nov 01 10:06:11.341346 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.341307 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.341531 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.341388 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.341531 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.341448 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpn4t\" (UniqueName: \"kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.342249 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.342171 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.344105 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.344079 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.399976 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.399941 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpn4t\" (UniqueName: \"kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t\") pod \"maas-api-revoke-sub-6b19338f-9ghfb\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.516707 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.516615 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:11.891391 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:11.891354 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb"] Nov 01 10:06:11.895142 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:06:11.895113 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508714a0_e9d9_4f79_9c72_650cbc2692a0.slice/crio-639e7f9f32b82223bb0389e49cd3a31187bdba384cbea86ace1751427c3c86c1 WatchSource:0}: Error finding container 639e7f9f32b82223bb0389e49cd3a31187bdba384cbea86ace1751427c3c86c1: Status 404 returned error can't find the container with id 639e7f9f32b82223bb0389e49cd3a31187bdba384cbea86ace1751427c3c86c1 Nov 01 10:06:12.356568 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:12.356532 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" event={"ID":"508714a0-e9d9-4f79-9c72-650cbc2692a0","Type":"ContainerStarted","Data":"ba625ffd0ede153949366355d4e5a8f227c20ba02fdb3562b728fa25f373b4c8"} Nov 01 10:06:12.356996 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:12.356589 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" event={"ID":"508714a0-e9d9-4f79-9c72-650cbc2692a0","Type":"ContainerStarted","Data":"639e7f9f32b82223bb0389e49cd3a31187bdba384cbea86ace1751427c3c86c1"} Nov 01 10:06:12.467996 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:12.467889 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" podStartSLOduration=1.467873784 podStartE2EDuration="1.467873784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:06:12.466396267 +0000 UTC m=+5595.554732940" watchObservedRunningTime="2026-11-01 10:06:12.467873784 +0000 UTC m=+5595.556210456" Nov 01 10:06:13.363655 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:13.363617 5710 generic.go:356] "Generic (PLEG): container finished" podID="508714a0-e9d9-4f79-9c72-650cbc2692a0" containerID="ba625ffd0ede153949366355d4e5a8f227c20ba02fdb3562b728fa25f373b4c8" exitCode=0 Nov 01 10:06:13.364112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:13.363669 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" event={"ID":"508714a0-e9d9-4f79-9c72-650cbc2692a0","Type":"ContainerDied","Data":"ba625ffd0ede153949366355d4e5a8f227c20ba02fdb3562b728fa25f373b4c8"} Nov 01 10:06:14.510899 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.510870 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:14.583286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.583251 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t\" (UniqueName: \"kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t\") pod \"508714a0-e9d9-4f79-9c72-650cbc2692a0\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " Nov 01 10:06:14.583484 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.583319 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca\") pod \"508714a0-e9d9-4f79-9c72-650cbc2692a0\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " Nov 01 10:06:14.583484 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.583389 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca\") pod \"508714a0-e9d9-4f79-9c72-650cbc2692a0\" (UID: \"508714a0-e9d9-4f79-9c72-650cbc2692a0\") " Nov 01 10:06:14.583840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.583807 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca" pod "508714a0-e9d9-4f79-9c72-650cbc2692a0" (UID: "508714a0-e9d9-4f79-9c72-650cbc2692a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:06:14.585626 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.585604 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca" pod "508714a0-e9d9-4f79-9c72-650cbc2692a0" (UID: "508714a0-e9d9-4f79-9c72-650cbc2692a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:06:14.585911 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.585890 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t" pod "508714a0-e9d9-4f79-9c72-650cbc2692a0" (UID: "508714a0-e9d9-4f79-9c72-650cbc2692a0"). InnerVolumeSpecName "kube-api-access-mpn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:06:14.684521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.684429 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpn4t\" (UniqueName: \"kubernetes.io/projected/508714a0-e9d9-4f79-9c72-650cbc2692a0-kube-api-access-mpn4t\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:06:14.684521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.684459 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:06:14.684521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:14.684471 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508714a0-e9d9-4f79-9c72-650cbc2692a0-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:06:15.374428 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:15.374400 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" Nov 01 10:06:15.374624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:15.374442 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb" event={"ID":"508714a0-e9d9-4f79-9c72-650cbc2692a0","Type":"ContainerDied","Data":"639e7f9f32b82223bb0389e49cd3a31187bdba384cbea86ace1751427c3c86c1"} Nov 01 10:06:15.374624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:15.374490 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="639e7f9f32b82223bb0389e49cd3a31187bdba384cbea86ace1751427c3c86c1" Nov 01 10:06:15.476853 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:06:15.476817 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:06:23.954917 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:23.954883 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:06:26.503746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.502857 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8"] Nov 01 10:06:26.503746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.503590 5710 cpu_manager.go:414] "containerMap: removing container" podUID="508714a0-e9d9-4f79-9c72-650cbc2692a0" containerName="revoke-keys" Nov 01 10:06:26.503746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.503614 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508714a0-e9d9-4f79-9c72-650cbc2692a0" containerName="revoke-keys" Nov 01 10:06:26.526746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.524794 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="508714a0-e9d9-4f79-9c72-650cbc2692a0" containerName="revoke-keys" Nov 01 10:06:26.533745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.529045 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.565773 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.565631 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8"] Nov 01 10:06:26.620747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.620078 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-d609fa82-kserve-self-signed-certs\"" Nov 01 10:06:26.620747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.620170 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-d609fa82-epp-sa-dockercfg-p85mq\"" Nov 01 10:06:26.656560 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.656265 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs\") pod \"e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.656560 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.656360 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrkk\" (UniqueName: \"kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk\") pod \"e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.759500 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.758243 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs\") pod \"e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.759500 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.758332 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrkk\" (UniqueName: \"kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk\") pod \"e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.765744 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.765684 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs\") pod \"e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.818343 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.818292 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrkk\" (UniqueName: \"kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk\") pod \"e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:26.862285 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:26.862241 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:27.250733 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:27.250678 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8"] Nov 01 10:06:27.253975 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:06:27.253943 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cbe801_880d_418b_bc7f_1684592208b4.slice/crio-75b45d82c33e452532f3c232267fed17f671e8999dbf85d55b4573a95f36fdbf WatchSource:0}: Error finding container 75b45d82c33e452532f3c232267fed17f671e8999dbf85d55b4573a95f36fdbf: Status 404 returned error can't find the container with id 75b45d82c33e452532f3c232267fed17f671e8999dbf85d55b4573a95f36fdbf Nov 01 10:06:27.460807 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:27.460771 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" event={"ID":"43cbe801-880d-418b-bc7f-1684592208b4","Type":"ContainerStarted","Data":"75b45d82c33e452532f3c232267fed17f671e8999dbf85d55b4573a95f36fdbf"} Nov 01 10:06:30.477280 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:30.477217 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" event={"ID":"43cbe801-880d-418b-bc7f-1684592208b4","Type":"ContainerStarted","Data":"733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2"} Nov 01 10:06:30.477829 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:30.477306 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:06:34.904959 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:34.904875 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:06:34.905678 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:34.905643 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:06:37.288344 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:37.288285 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" podStartSLOduration=9.121364297 podStartE2EDuration="11.288264974s" totalImagesPullingTime="2.166900677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:06:30.588912705 +0000 UTC m=+5613.677249377" watchObservedRunningTime="2026-11-01 10:06:37.288264974 +0000 UTC m=+5620.376601645" Nov 01 10:06:37.289211 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:06:37.289192 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:06:45.477638 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:06:45.477599 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:07:01.487251 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:01.487210 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:07:09.755613 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:09.755570 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:07:12.657854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:12.655904 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:07:12.657854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:12.656034 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:07:15.477693 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:07:15.477655 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:07:35.085230 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:35.085147 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:07:35.208503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:35.208458 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8"] Nov 01 10:07:41.486967 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:41.486930 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:07:45.479241 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:07:45.479178 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:07:47.676211 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:47.676175 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:07:50.231513 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:50.231454 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" containerID="cri-o://733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2" gracePeriod=45 Nov 01 10:07:51.487354 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:07:51.487310 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:08:01.438288 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:01.438256 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:08:01.450609 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:01.450581 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:08:01.486567 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:01.486528 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:08:01.486792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:01.486646 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:08:11.487592 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:11.487546 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:08:11.661580 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:11.661538 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:08:15.476646 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:08:15.476609 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:08:20.501605 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.501514 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:08:20.602506 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.602473 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk\" (UniqueName: \"kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk\") pod \"43cbe801-880d-418b-bc7f-1684592208b4\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " Nov 01 10:08:20.602764 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.602535 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs\") pod \"43cbe801-880d-418b-bc7f-1684592208b4\" (UID: \"43cbe801-880d-418b-bc7f-1684592208b4\") " Nov 01 10:08:20.604997 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.604969 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs" pod "43cbe801-880d-418b-bc7f-1684592208b4" (UID: "43cbe801-880d-418b-bc7f-1684592208b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:08:20.605143 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.605019 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk" pod "43cbe801-880d-418b-bc7f-1684592208b4" (UID: "43cbe801-880d-418b-bc7f-1684592208b4"). InnerVolumeSpecName "kube-api-access-7qrkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:08:20.704016 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.703964 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qrkk\" (UniqueName: \"kubernetes.io/projected/43cbe801-880d-418b-bc7f-1684592208b4-kube-api-access-7qrkk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:08:20.704016 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:20.704012 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43cbe801-880d-418b-bc7f-1684592208b4-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:08:21.197818 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.197784 5710 generic.go:356] "Generic (PLEG): container finished" podID="43cbe801-880d-418b-bc7f-1684592208b4" containerID="733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2" exitCode=0 Nov 01 10:08:21.198022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.197833 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" Nov 01 10:08:21.198022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.197879 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" event={"ID":"43cbe801-880d-418b-bc7f-1684592208b4","Type":"ContainerDied","Data":"733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2"} Nov 01 10:08:21.198022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.197940 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8" event={"ID":"43cbe801-880d-418b-bc7f-1684592208b4","Type":"ContainerDied","Data":"75b45d82c33e452532f3c232267fed17f671e8999dbf85d55b4573a95f36fdbf"} Nov 01 10:08:21.198022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.197971 5710 scope.go:122] "RemoveContainer" containerID="733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2" Nov 01 10:08:21.206988 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.206967 5710 scope.go:122] "RemoveContainer" containerID="733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2" Nov 01 10:08:21.207313 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:08:21.207289 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2\": container with ID starting with 733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2 not found: ID does not exist" containerID="733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2" Nov 01 10:08:21.207363 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.207326 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2"} err="failed to get container status \"733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2\": rpc error: code = NotFound desc = could not find container \"733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2\": container with ID starting with 733d814a5555fb84f4b358fc58ed8aaba9acb03870e68754b6ce32e06250a4e2 not found: ID does not exist" Nov 01 10:08:21.355912 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.355873 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8"] Nov 01 10:08:21.402999 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:21.402959 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-d609fa82-kserve-router-scheduler-5cf795546d-m9tk8"] Nov 01 10:08:23.057039 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:23.057000 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cbe801-880d-418b-bc7f-1684592208b4" path="/var/lib/kubelet/pods/43cbe801-880d-418b-bc7f-1684592208b4/volumes" Nov 01 10:08:26.738011 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.737973 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6"] Nov 01 10:08:26.738463 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.738449 5710 cpu_manager.go:414] "containerMap: removing container" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" Nov 01 10:08:26.738508 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.738465 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" Nov 01 10:08:26.738553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.738540 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cbe801-880d-418b-bc7f-1684592208b4" containerName="main" Nov 01 10:08:26.743291 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.743266 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.794565 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.794536 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-029e39\"/\"kube-root-ca.crt\"" Nov 01 10:08:26.797399 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.797366 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6"] Nov 01 10:08:26.838405 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.838378 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-029e39\"/\"openshift-service-ca.crt\"" Nov 01 10:08:26.841448 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.841429 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-029e39\"/\"simulator-single-main-029e39-kserve-self-signed-certs\"" Nov 01 10:08:26.841695 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.841674 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-029e39\"/\"default-dockercfg-bkhx8\"" Nov 01 10:08:26.874014 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.873975 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.874238 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.874160 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjsz\" (UniqueName: \"kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.874238 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.874203 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.874379 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.874244 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.874379 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.874278 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.874379 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.874313 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975180 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975130 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjsz\" (UniqueName: \"kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975206 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975259 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975296 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975324 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975429 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975404 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975867 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975838 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975982 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975872 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.975982 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.975900 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.977875 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.977857 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:26.978256 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:26.978232 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:27.033238 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:27.033156 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjsz\" (UniqueName: \"kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz\") pod \"simulator-single-main-029e39-kserve-79d45c8466-tw5p6\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:27.055901 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:27.055863 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:27.427706 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:27.427655 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6"] Nov 01 10:08:27.431694 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:08:27.431660 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eda31ee_d868_43b7_9876_ac875cbc9271.slice/crio-f92e0ba20809c2797f36ea410cd073f340ebc6205c58730da28eb0430a7b2920 WatchSource:0}: Error finding container f92e0ba20809c2797f36ea410cd073f340ebc6205c58730da28eb0430a7b2920: Status 404 returned error can't find the container with id f92e0ba20809c2797f36ea410cd073f340ebc6205c58730da28eb0430a7b2920 Nov 01 10:08:28.237745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:28.237698 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" event={"ID":"9eda31ee-d868-43b7-9876-ac875cbc9271","Type":"ContainerStarted","Data":"6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4"} Nov 01 10:08:28.238168 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:28.237757 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" event={"ID":"9eda31ee-d868-43b7-9876-ac875cbc9271","Type":"ContainerStarted","Data":"f92e0ba20809c2797f36ea410cd073f340ebc6205c58730da28eb0430a7b2920"} Nov 01 10:08:28.348503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:28.348439 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" podStartSLOduration=2.348421613 podStartE2EDuration="2.348421613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:08:28.347785523 +0000 UTC m=+5731.436122195" watchObservedRunningTime="2026-11-01 10:08:28.348421613 +0000 UTC m=+5731.436758288" Nov 01 10:08:29.057254 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:29.057224 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:31.254402 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:31.254363 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:08:37.059851 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:37.059822 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:37.069564 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:37.069532 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:37.299062 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:37.299031 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:08:45.477491 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:08:45.477450 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:08:59.872986 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:08:59.872943 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:09:13.097590 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.097547 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl"] Nov 01 10:09:13.101340 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.101317 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.155921 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.155878 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.156269 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.155917 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl"] Nov 01 10:09:13.156414 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.156326 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2924r\" (UniqueName: \"kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.156414 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.156393 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.199592 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.199556 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:09:13.199592 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.199585 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:09:13.199961 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.199653 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:09:13.199961 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.199818 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:09:13.257278 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.257204 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2924r\" (UniqueName: \"kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.257547 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.257295 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.257547 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.257337 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.258276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.258247 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.260317 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.260299 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.315930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.315898 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2924r\" (UniqueName: \"kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r\") pod \"maas-api-revoke-sub-172546e2-pjszl\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.415147 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.415059 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:13.789117 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:13.788910 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl"] Nov 01 10:09:13.792388 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:09:13.792343 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80c78da_1674_4453_9021_f37d0d5e7803.slice/crio-161bcdacf97652e266e0712ad42c5be3a83e4faf75b5b0d004094ae275a65208 WatchSource:0}: Error finding container 161bcdacf97652e266e0712ad42c5be3a83e4faf75b5b0d004094ae275a65208: Status 404 returned error can't find the container with id 161bcdacf97652e266e0712ad42c5be3a83e4faf75b5b0d004094ae275a65208 Nov 01 10:09:14.485234 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:14.485139 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" event={"ID":"a80c78da-1674-4453-9021-f37d0d5e7803","Type":"ContainerStarted","Data":"4ff4f7be5d84187e30c220149979c7eac0bdd58d042c85704748ccaac7d4d43f"} Nov 01 10:09:14.485234 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:14.485195 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" event={"ID":"a80c78da-1674-4453-9021-f37d0d5e7803","Type":"ContainerStarted","Data":"161bcdacf97652e266e0712ad42c5be3a83e4faf75b5b0d004094ae275a65208"} Nov 01 10:09:14.594827 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:14.594771 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" podStartSLOduration=1.594753251 podStartE2EDuration="1.594753251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:09:14.594184391 +0000 UTC m=+5777.682521063" watchObservedRunningTime="2026-11-01 10:09:14.594753251 +0000 UTC m=+5777.683089922" Nov 01 10:09:15.479658 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:09:15.479613 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:09:15.493104 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:15.493074 5710 generic.go:356] "Generic (PLEG): container finished" podID="a80c78da-1674-4453-9021-f37d0d5e7803" containerID="4ff4f7be5d84187e30c220149979c7eac0bdd58d042c85704748ccaac7d4d43f" exitCode=0 Nov 01 10:09:15.493516 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:15.493116 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" event={"ID":"a80c78da-1674-4453-9021-f37d0d5e7803","Type":"ContainerDied","Data":"4ff4f7be5d84187e30c220149979c7eac0bdd58d042c85704748ccaac7d4d43f"} Nov 01 10:09:16.650652 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.650622 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:16.691972 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.691946 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca\") pod \"a80c78da-1674-4453-9021-f37d0d5e7803\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " Nov 01 10:09:16.692189 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.692061 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca\") pod \"a80c78da-1674-4453-9021-f37d0d5e7803\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " Nov 01 10:09:16.692189 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.692097 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r\" (UniqueName: \"kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r\") pod \"a80c78da-1674-4453-9021-f37d0d5e7803\" (UID: \"a80c78da-1674-4453-9021-f37d0d5e7803\") " Nov 01 10:09:16.692441 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.692412 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca" pod "a80c78da-1674-4453-9021-f37d0d5e7803" (UID: "a80c78da-1674-4453-9021-f37d0d5e7803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:09:16.694368 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.694345 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca" pod "a80c78da-1674-4453-9021-f37d0d5e7803" (UID: "a80c78da-1674-4453-9021-f37d0d5e7803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:09:16.694475 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.694405 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r" pod "a80c78da-1674-4453-9021-f37d0d5e7803" (UID: "a80c78da-1674-4453-9021-f37d0d5e7803"). InnerVolumeSpecName "kube-api-access-2924r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:09:16.793654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.793564 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:16.793654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.793598 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80c78da-1674-4453-9021-f37d0d5e7803-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:16.793654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:16.793610 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2924r\" (UniqueName: \"kubernetes.io/projected/a80c78da-1674-4453-9021-f37d0d5e7803-kube-api-access-2924r\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:17.506007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:17.505977 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" Nov 01 10:09:17.506199 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:17.505976 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl" event={"ID":"a80c78da-1674-4453-9021-f37d0d5e7803","Type":"ContainerDied","Data":"161bcdacf97652e266e0712ad42c5be3a83e4faf75b5b0d004094ae275a65208"} Nov 01 10:09:17.506199 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:17.506104 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161bcdacf97652e266e0712ad42c5be3a83e4faf75b5b0d004094ae275a65208" Nov 01 10:09:25.992074 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:25.992033 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:09:29.160503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:29.160467 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6"] Nov 01 10:09:40.588396 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:40.588363 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:09:44.181658 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.181620 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" podUID="9eda31ee-d868-43b7-9876-ac875cbc9271" containerName="main" containerID="cri-o://6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4" gracePeriod=45 Nov 01 10:09:44.288094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.288054 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" podUID="9eda31ee-d868-43b7-9876-ac875cbc9271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.250:8000/ready\": dial tcp 10.134.0.250:8000: connect: connection refused" Nov 01 10:09:44.443319 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.443250 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:09:44.499043 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499004 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir\") pod \"9eda31ee-d868-43b7-9876-ac875cbc9271\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " Nov 01 10:09:44.499276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499066 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home\") pod \"9eda31ee-d868-43b7-9876-ac875cbc9271\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " Nov 01 10:09:44.499276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499099 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache\") pod \"9eda31ee-d868-43b7-9876-ac875cbc9271\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " Nov 01 10:09:44.499276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499123 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm\") pod \"9eda31ee-d868-43b7-9876-ac875cbc9271\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " Nov 01 10:09:44.499276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499168 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz\" (UniqueName: \"kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz\") pod \"9eda31ee-d868-43b7-9876-ac875cbc9271\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " Nov 01 10:09:44.499276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499243 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs\") pod \"9eda31ee-d868-43b7-9876-ac875cbc9271\" (UID: \"9eda31ee-d868-43b7-9876-ac875cbc9271\") " Nov 01 10:09:44.499579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499356 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home" pod "9eda31ee-d868-43b7-9876-ac875cbc9271" (UID: "9eda31ee-d868-43b7-9876-ac875cbc9271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:09:44.499579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499372 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir" pod "9eda31ee-d868-43b7-9876-ac875cbc9271" (UID: "9eda31ee-d868-43b7-9876-ac875cbc9271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:09:44.499579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499391 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache" pod "9eda31ee-d868-43b7-9876-ac875cbc9271" (UID: "9eda31ee-d868-43b7-9876-ac875cbc9271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:09:44.499762 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499614 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:44.499762 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499630 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:44.499762 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.499640 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:44.501594 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.501567 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm" pod "9eda31ee-d868-43b7-9876-ac875cbc9271" (UID: "9eda31ee-d868-43b7-9876-ac875cbc9271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:09:44.501737 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.501650 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs" pod "9eda31ee-d868-43b7-9876-ac875cbc9271" (UID: "9eda31ee-d868-43b7-9876-ac875cbc9271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:09:44.501737 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.501654 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz" pod "9eda31ee-d868-43b7-9876-ac875cbc9271" (UID: "9eda31ee-d868-43b7-9876-ac875cbc9271"). InnerVolumeSpecName "kube-api-access-2hjsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:09:44.600956 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.600922 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eda31ee-d868-43b7-9876-ac875cbc9271-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:44.600956 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.600953 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjsz\" (UniqueName: \"kubernetes.io/projected/9eda31ee-d868-43b7-9876-ac875cbc9271-kube-api-access-2hjsz\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:44.600956 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.600963 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eda31ee-d868-43b7-9876-ac875cbc9271-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:09:44.671329 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.671287 5710 generic.go:356] "Generic (PLEG): container finished" podID="9eda31ee-d868-43b7-9876-ac875cbc9271" containerID="6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4" exitCode=0 Nov 01 10:09:44.671521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.671422 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" event={"ID":"9eda31ee-d868-43b7-9876-ac875cbc9271","Type":"ContainerDied","Data":"6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4"} Nov 01 10:09:44.671521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.671428 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" Nov 01 10:09:44.671521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.671459 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6" event={"ID":"9eda31ee-d868-43b7-9876-ac875cbc9271","Type":"ContainerDied","Data":"f92e0ba20809c2797f36ea410cd073f340ebc6205c58730da28eb0430a7b2920"} Nov 01 10:09:44.671638 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.671579 5710 scope.go:122] "RemoveContainer" containerID="6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4" Nov 01 10:09:44.682125 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.682100 5710 scope.go:122] "RemoveContainer" containerID="6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4" Nov 01 10:09:44.682399 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:09:44.682377 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4\": container with ID starting with 6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4 not found: ID does not exist" containerID="6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4" Nov 01 10:09:44.682453 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.682406 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4"} err="failed to get container status \"6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4\": rpc error: code = NotFound desc = could not find container \"6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4\": container with ID starting with 6dd6c8d12e679658d05ac5b1eba83a89bca7b60956266750f409fa0fe95b0bb4 not found: ID does not exist" Nov 01 10:09:44.826349 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.826259 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6"] Nov 01 10:09:44.830139 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:44.830110 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-029e39/simulator-single-main-029e39-kserve-79d45c8466-tw5p6"] Nov 01 10:09:45.057350 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:09:45.057311 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eda31ee-d868-43b7-9876-ac875cbc9271" path="/var/lib/kubelet/pods/9eda31ee-d868-43b7-9876-ac875cbc9271/volumes" Nov 01 10:09:45.477798 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:09:45.477761 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:10:11.066386 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:11.066349 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl"] Nov 01 10:10:11.112481 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:11.112445 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-wcgzl"] Nov 01 10:10:13.057235 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:13.057188 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f27c92-a19c-4af4-bea4-ae948efae41c" path="/var/lib/kubelet/pods/e5f27c92-a19c-4af4-bea4-ae948efae41c/volumes" Nov 01 10:10:15.476593 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:10:15.476551 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:10:30.864926 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:30.864875 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:10:34.899953 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:34.899852 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:10:34.900789 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:34.900749 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:10:44.047302 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:44.047261 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n"] Nov 01 10:10:44.093640 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:44.093602 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6999n"] Nov 01 10:10:45.057464 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:45.057422 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800feaee-0811-482b-a561-8eb3b53f4266" path="/var/lib/kubelet/pods/800feaee-0811-482b-a561-8eb3b53f4266/volumes" Nov 01 10:10:45.482741 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:10:45.482642 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:10:53.732022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.731984 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx"] Nov 01 10:10:53.732553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.732395 5710 cpu_manager.go:414] "containerMap: removing container" podUID="a80c78da-1674-4453-9021-f37d0d5e7803" containerName="revoke-keys" Nov 01 10:10:53.732553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.732408 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80c78da-1674-4453-9021-f37d0d5e7803" containerName="revoke-keys" Nov 01 10:10:53.732553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.732428 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9eda31ee-d868-43b7-9876-ac875cbc9271" containerName="main" Nov 01 10:10:53.732553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.732433 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eda31ee-d868-43b7-9876-ac875cbc9271" containerName="main" Nov 01 10:10:53.732553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.732512 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80c78da-1674-4453-9021-f37d0d5e7803" containerName="revoke-keys" Nov 01 10:10:53.732553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.732525 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eda31ee-d868-43b7-9876-ac875cbc9271" containerName="main" Nov 01 10:10:53.735934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.735910 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.831224 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.831179 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-46760c40-kserve-self-signed-certs\"" Nov 01 10:10:53.833105 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.833082 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx"] Nov 01 10:10:53.872932 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.872890 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.873163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.872954 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.873163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.873021 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.873163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.873049 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.873163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.873076 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.873163 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.873118 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974190 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974154 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974190 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974191 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974209 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974244 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974354 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974392 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974703 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974753 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.974934 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.974790 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.976620 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.976602 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:53.977079 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:53.977059 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:54.033129 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:54.033063 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2\") pod \"e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:54.049899 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:54.049875 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:54.445104 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:54.444882 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx"] Nov 01 10:10:54.448046 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:10:54.448011 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode47d3b59_86c4_4a95_acc1_e602b39553f1.slice/crio-e6daca5c7c8ff77b042a66ea33aebf6586d765ea170cf26bdb791526f33e884e WatchSource:0}: Error finding container e6daca5c7c8ff77b042a66ea33aebf6586d765ea170cf26bdb791526f33e884e: Status 404 returned error can't find the container with id e6daca5c7c8ff77b042a66ea33aebf6586d765ea170cf26bdb791526f33e884e Nov 01 10:10:55.068679 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:55.068637 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" event={"ID":"e47d3b59-86c4-4a95-acc1-e602b39553f1","Type":"ContainerStarted","Data":"baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d"} Nov 01 10:10:55.069128 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:55.068690 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" event={"ID":"e47d3b59-86c4-4a95-acc1-e602b39553f1","Type":"ContainerStarted","Data":"e6daca5c7c8ff77b042a66ea33aebf6586d765ea170cf26bdb791526f33e884e"} Nov 01 10:10:55.179592 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:55.179527 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" podStartSLOduration=2.179508278 podStartE2EDuration="2.179508278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:10:55.179180797 +0000 UTC m=+5878.267517470" watchObservedRunningTime="2026-11-01 10:10:55.179508278 +0000 UTC m=+5878.267844955" Nov 01 10:10:56.050445 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:56.050393 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:10:58.314146 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:10:58.314104 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:11:03.048084 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:03.047959 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht"] Nov 01 10:11:03.113746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:03.094541 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lplht"] Nov 01 10:11:04.050782 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:04.050743 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:11:04.063657 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:04.063627 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:11:04.178123 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:04.178090 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:11:05.057531 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:05.057493 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7710ad5f-b40e-436c-a12f-acc51da9319e" path="/var/lib/kubelet/pods/7710ad5f-b40e-436c-a12f-acc51da9319e/volumes" Nov 01 10:11:15.057118 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:15.057086 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb"] Nov 01 10:11:15.105434 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:15.105391 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ghfb"] Nov 01 10:11:15.478049 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:11:15.478007 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:11:17.058823 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:17.058791 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508714a0-e9d9-4f79-9c72-650cbc2692a0" path="/var/lib/kubelet/pods/508714a0-e9d9-4f79-9c72-650cbc2692a0/volumes" Nov 01 10:11:28.494654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:28.494615 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:11:42.153982 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:42.153942 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:11:45.477680 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:11:45.477643 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:11:48.529023 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:48.528982 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx"] Nov 01 10:11:53.379975 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:11:53.379937 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:12:01.158132 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:01.158097 5710 scope.go:122] "RemoveContainer" containerID="dc02e1547de484737ce4a88ce01047b16a5a2ee60e30985465d0c45384ca344e" Nov 01 10:12:01.170205 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:01.170182 5710 scope.go:122] "RemoveContainer" containerID="672f56b0e2904980b171bcd331ff6dec1be25c3ce83854a8885ed844c8ec4551" Nov 01 10:12:01.179792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:01.179768 5710 scope.go:122] "RemoveContainer" containerID="f2ae477500663c9a077e08991db58466a3b8d32cb201f3ae64600bac803a6427" Nov 01 10:12:03.551606 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.551533 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" podUID="e47d3b59-86c4-4a95-acc1-e602b39553f1" containerName="main" containerID="cri-o://baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d" gracePeriod=45 Nov 01 10:12:03.810476 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.810405 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:12:03.937820 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.937776 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home\") pod \"e47d3b59-86c4-4a95-acc1-e602b39553f1\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " Nov 01 10:12:03.938063 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.937860 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs\") pod \"e47d3b59-86c4-4a95-acc1-e602b39553f1\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " Nov 01 10:12:03.938063 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938049 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir\") pod \"e47d3b59-86c4-4a95-acc1-e602b39553f1\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " Nov 01 10:12:03.938223 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938088 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache\") pod \"e47d3b59-86c4-4a95-acc1-e602b39553f1\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " Nov 01 10:12:03.938223 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938092 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home" pod "e47d3b59-86c4-4a95-acc1-e602b39553f1" (UID: "e47d3b59-86c4-4a95-acc1-e602b39553f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:12:03.938223 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938140 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm\") pod \"e47d3b59-86c4-4a95-acc1-e602b39553f1\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " Nov 01 10:12:03.938223 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938208 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2\") pod \"e47d3b59-86c4-4a95-acc1-e602b39553f1\" (UID: \"e47d3b59-86c4-4a95-acc1-e602b39553f1\") " Nov 01 10:12:03.938457 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938318 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir" pod "e47d3b59-86c4-4a95-acc1-e602b39553f1" (UID: "e47d3b59-86c4-4a95-acc1-e602b39553f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:12:03.938457 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938359 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache" pod "e47d3b59-86c4-4a95-acc1-e602b39553f1" (UID: "e47d3b59-86c4-4a95-acc1-e602b39553f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:12:03.938930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938833 5710 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-tmp-dir\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:03.938930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938864 5710 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-model-cache\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:03.938930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.938883 5710 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-home\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:03.940357 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.940323 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs" pod "e47d3b59-86c4-4a95-acc1-e602b39553f1" (UID: "e47d3b59-86c4-4a95-acc1-e602b39553f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:12:03.940795 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.940753 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm" pod "e47d3b59-86c4-4a95-acc1-e602b39553f1" (UID: "e47d3b59-86c4-4a95-acc1-e602b39553f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:12:03.941061 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:03.941040 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2" pod "e47d3b59-86c4-4a95-acc1-e602b39553f1" (UID: "e47d3b59-86c4-4a95-acc1-e602b39553f1"). InnerVolumeSpecName "kube-api-access-hknm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:12:04.039896 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.039847 5710 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d3b59-86c4-4a95-acc1-e602b39553f1-dshm\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:04.039896 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.039892 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/e47d3b59-86c4-4a95-acc1-e602b39553f1-kube-api-access-hknm2\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:04.040182 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.039908 5710 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d3b59-86c4-4a95-acc1-e602b39553f1-tls-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:04.579486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.579394 5710 generic.go:356] "Generic (PLEG): container finished" podID="e47d3b59-86c4-4a95-acc1-e602b39553f1" containerID="baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d" exitCode=0 Nov 01 10:12:04.579486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.579456 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" Nov 01 10:12:04.580086 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.579490 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" event={"ID":"e47d3b59-86c4-4a95-acc1-e602b39553f1","Type":"ContainerDied","Data":"baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d"} Nov 01 10:12:04.580086 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.579547 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx" event={"ID":"e47d3b59-86c4-4a95-acc1-e602b39553f1","Type":"ContainerDied","Data":"e6daca5c7c8ff77b042a66ea33aebf6586d765ea170cf26bdb791526f33e884e"} Nov 01 10:12:04.580086 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.579572 5710 scope.go:122] "RemoveContainer" containerID="baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d" Nov 01 10:12:04.590905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.590885 5710 scope.go:122] "RemoveContainer" containerID="baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d" Nov 01 10:12:04.591240 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:12:04.591211 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d\": container with ID starting with baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d not found: ID does not exist" containerID="baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d" Nov 01 10:12:04.591316 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.591250 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d"} err="failed to get container status \"baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d\": rpc error: code = NotFound desc = could not find container \"baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d\": container with ID starting with baffe0438a34e5e8217a04c13c2b757bb23045ea8661c79068c63f3e8ec1eb0d not found: ID does not exist" Nov 01 10:12:04.748651 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.748609 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx"] Nov 01 10:12:04.752971 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:04.752941 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-46760c40-kserve-6dc8b88fd9-h9scx"] Nov 01 10:12:05.059687 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:05.059011 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47d3b59-86c4-4a95-acc1-e602b39553f1" path="/var/lib/kubelet/pods/e47d3b59-86c4-4a95-acc1-e602b39553f1/volumes" Nov 01 10:12:06.452620 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:06.452582 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:12:09.353901 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:09.353861 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:12:09.354545 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:09.354512 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:12:14.048310 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:14.048270 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:12:15.553243 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:12:15.553200 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:12:20.061394 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.061362 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc"] Nov 01 10:12:20.061993 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.061973 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e47d3b59-86c4-4a95-acc1-e602b39553f1" containerName="main" Nov 01 10:12:20.061993 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.061992 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47d3b59-86c4-4a95-acc1-e602b39553f1" containerName="main" Nov 01 10:12:20.062158 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.062091 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47d3b59-86c4-4a95-acc1-e602b39553f1" containerName="main" Nov 01 10:12:20.065558 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.065530 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.084751 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.084708 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.084945 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.084816 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.085026 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.084943 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx8nq\" (UniqueName: \"kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.119447 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.119408 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc"] Nov 01 10:12:20.169304 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.169270 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:12:20.169304 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.169270 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:12:20.169568 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.169270 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:12:20.171602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.171583 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:12:20.185958 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.185927 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx8nq\" (UniqueName: \"kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.186052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.185978 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.186052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.186030 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.186765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.186746 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.188580 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.188556 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.249751 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.249682 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx8nq\" (UniqueName: \"kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq\") pod \"maas-api-revoke-sub-14fd1ea7-xn6wc\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.381299 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.381257 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:20.755745 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:12:20.755693 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033aa63e_7d21_4968_8d17_fb6ebc95c770.slice/crio-1cada355a0ab4541be7390a49b1bcfc66fe1eeffe4250ee85dbe43af07291876 WatchSource:0}: Error finding container 1cada355a0ab4541be7390a49b1bcfc66fe1eeffe4250ee85dbe43af07291876: Status 404 returned error can't find the container with id 1cada355a0ab4541be7390a49b1bcfc66fe1eeffe4250ee85dbe43af07291876 Nov 01 10:12:20.756778 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:20.756704 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc"] Nov 01 10:12:21.709369 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:21.709336 5710 generic.go:356] "Generic (PLEG): container finished" podID="033aa63e-7d21-4968-8d17-fb6ebc95c770" containerID="21065f2dedf0597ef203575946595b6862c520f2aa0e8afe705fe78bc053dc89" exitCode=0 Nov 01 10:12:21.709369 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:21.709374 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" event={"ID":"033aa63e-7d21-4968-8d17-fb6ebc95c770","Type":"ContainerDied","Data":"21065f2dedf0597ef203575946595b6862c520f2aa0e8afe705fe78bc053dc89"} Nov 01 10:12:21.709887 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:21.709413 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" event={"ID":"033aa63e-7d21-4968-8d17-fb6ebc95c770","Type":"ContainerStarted","Data":"1cada355a0ab4541be7390a49b1bcfc66fe1eeffe4250ee85dbe43af07291876"} Nov 01 10:12:22.860555 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.860525 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:22.911571 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.911529 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq\" (UniqueName: \"kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq\") pod \"033aa63e-7d21-4968-8d17-fb6ebc95c770\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " Nov 01 10:12:22.911834 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.911643 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca\") pod \"033aa63e-7d21-4968-8d17-fb6ebc95c770\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " Nov 01 10:12:22.911834 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.911769 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca\") pod \"033aa63e-7d21-4968-8d17-fb6ebc95c770\" (UID: \"033aa63e-7d21-4968-8d17-fb6ebc95c770\") " Nov 01 10:12:22.912109 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.912084 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca" pod "033aa63e-7d21-4968-8d17-fb6ebc95c770" (UID: "033aa63e-7d21-4968-8d17-fb6ebc95c770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:12:22.912191 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.912158 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:22.914096 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.914068 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq" pod "033aa63e-7d21-4968-8d17-fb6ebc95c770" (UID: "033aa63e-7d21-4968-8d17-fb6ebc95c770"). InnerVolumeSpecName "kube-api-access-wx8nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:12:22.914210 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:22.914159 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca" pod "033aa63e-7d21-4968-8d17-fb6ebc95c770" (UID: "033aa63e-7d21-4968-8d17-fb6ebc95c770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:12:23.013091 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:23.013005 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx8nq\" (UniqueName: \"kubernetes.io/projected/033aa63e-7d21-4968-8d17-fb6ebc95c770-kube-api-access-wx8nq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:23.013091 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:23.013039 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033aa63e-7d21-4968-8d17-fb6ebc95c770-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:23.721416 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:23.721385 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" Nov 01 10:12:23.721662 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:23.721420 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc" event={"ID":"033aa63e-7d21-4968-8d17-fb6ebc95c770","Type":"ContainerDied","Data":"1cada355a0ab4541be7390a49b1bcfc66fe1eeffe4250ee85dbe43af07291876"} Nov 01 10:12:23.721662 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:23.721520 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cada355a0ab4541be7390a49b1bcfc66fe1eeffe4250ee85dbe43af07291876" Nov 01 10:12:32.692308 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:32.692270 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:12:36.663624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.663589 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t"] Nov 01 10:12:36.664134 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.664103 5710 cpu_manager.go:414] "containerMap: removing container" podUID="033aa63e-7d21-4968-8d17-fb6ebc95c770" containerName="revoke-keys" Nov 01 10:12:36.664134 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.664116 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033aa63e-7d21-4968-8d17-fb6ebc95c770" containerName="revoke-keys" Nov 01 10:12:36.664231 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.664211 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="033aa63e-7d21-4968-8d17-fb6ebc95c770" containerName="revoke-keys" Nov 01 10:12:36.667637 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.667617 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.723650 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.723614 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t"] Nov 01 10:12:36.755881 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.755839 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6527b\" (UniqueName: \"kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b\") pod \"payload-pre-processing-78d99cfb56-mbj9t\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.755881 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.755883 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume\") pod \"payload-pre-processing-78d99cfb56-mbj9t\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.857286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.857237 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6527b\" (UniqueName: \"kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b\") pod \"payload-pre-processing-78d99cfb56-mbj9t\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.857286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.857285 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume\") pod \"payload-pre-processing-78d99cfb56-mbj9t\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.858129 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.858107 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume\") pod \"payload-pre-processing-78d99cfb56-mbj9t\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.920944 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.920855 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6527b\" (UniqueName: \"kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b\") pod \"payload-pre-processing-78d99cfb56-mbj9t\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:36.981095 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:36.981055 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:37.355318 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:12:37.355273 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b67adf3_36f7_4990_9406_c163ea5bd45e.slice/crio-df1cb89ba2c55179dece5e2a0783e038b2759e54bb21c3fdcfa871728e2e6cce WatchSource:0}: Error finding container df1cb89ba2c55179dece5e2a0783e038b2759e54bb21c3fdcfa871728e2e6cce: Status 404 returned error can't find the container with id df1cb89ba2c55179dece5e2a0783e038b2759e54bb21c3fdcfa871728e2e6cce Nov 01 10:12:37.357340 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:37.357307 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t"] Nov 01 10:12:37.791761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:37.791635 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" event={"ID":"9b67adf3-36f7-4990-9406-c163ea5bd45e","Type":"ContainerStarted","Data":"856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c"} Nov 01 10:12:37.791761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:37.791701 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:37.791761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:37.791732 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" event={"ID":"9b67adf3-36f7-4990-9406-c163ea5bd45e","Type":"ContainerStarted","Data":"df1cb89ba2c55179dece5e2a0783e038b2759e54bb21c3fdcfa871728e2e6cce"} Nov 01 10:12:37.912488 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:37.912430 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" podStartSLOduration=1.91241692 podStartE2EDuration="1.91241692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:12:37.911627799 +0000 UTC m=+5980.999964472" watchObservedRunningTime="2026-11-01 10:12:37.91241692 +0000 UTC m=+5981.000753628" Nov 01 10:12:45.478463 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:12:45.478422 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:12:48.801937 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:48.801907 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:12:48.964051 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:48.964014 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6"] Nov 01 10:12:53.964516 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:53.964443 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" podUID="27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" containerName="payload-pre-processing" containerID="cri-o://747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe" gracePeriod=25 Nov 01 10:12:54.226760 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.226660 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 10:12:54.337395 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.337364 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume\") pod \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " Nov 01 10:12:54.337618 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.337484 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk\" (UniqueName: \"kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk\") pod \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\" (UID: \"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c\") " Nov 01 10:12:54.337777 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.337753 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume" pod "27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" (UID: "27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:12:54.337954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.337935 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:54.339871 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.339848 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk" pod "27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" (UID: "27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c"). InnerVolumeSpecName "kube-api-access-p5jsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:12:54.439510 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.439466 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5jsk\" (UniqueName: \"kubernetes.io/projected/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c-kube-api-access-p5jsk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:12:54.880554 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.880519 5710 generic.go:356] "Generic (PLEG): container finished" podID="27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" containerID="747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe" exitCode=0 Nov 01 10:12:54.880792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.880574 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" Nov 01 10:12:54.880792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.880577 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" event={"ID":"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c","Type":"ContainerDied","Data":"747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe"} Nov 01 10:12:54.880792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.880689 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6" event={"ID":"27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c","Type":"ContainerDied","Data":"6d5e75dc60d609c6eb303ee35d77c700e65de169d733069e45132ebe16e93ac7"} Nov 01 10:12:54.880792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.880710 5710 scope.go:122] "RemoveContainer" containerID="747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe" Nov 01 10:12:54.889862 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.889840 5710 scope.go:122] "RemoveContainer" containerID="747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe" Nov 01 10:12:54.890180 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:12:54.890159 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe\": container with ID starting with 747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe not found: ID does not exist" containerID="747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe" Nov 01 10:12:54.890246 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:54.890190 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe"} err="failed to get container status \"747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe\": rpc error: code = NotFound desc = could not find container \"747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe\": container with ID starting with 747b39ba119eb3ab2ef9ededbb385578e1027b271134c8eefe26f3aa8a849fbe not found: ID does not exist" Nov 01 10:12:55.065059 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:55.065026 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6"] Nov 01 10:12:55.065059 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:55.065065 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7hrd6"] Nov 01 10:12:57.056696 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:12:57.056661 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" path="/var/lib/kubelet/pods/27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c/volumes" Nov 01 10:13:01.209324 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:01.209290 5710 scope.go:122] "RemoveContainer" containerID="ba625ffd0ede153949366355d4e5a8f227c20ba02fdb3562b728fa25f373b4c8" Nov 01 10:13:01.471879 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:01.471795 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:13:01.484982 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:01.484955 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:13:08.427181 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:08.427141 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sccmk"] Nov 01 10:13:10.836019 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.835979 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl"] Nov 01 10:13:10.836448 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.836410 5710 cpu_manager.go:414] "containerMap: removing container" podUID="27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" containerName="payload-pre-processing" Nov 01 10:13:10.836448 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.836422 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" containerName="payload-pre-processing" Nov 01 10:13:10.836527 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.836512 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ef7fd8-4cd8-4e62-b4f9-45a690ea7b1c" containerName="payload-pre-processing" Nov 01 10:13:10.841342 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.841318 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:10.894475 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.894439 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl"] Nov 01 10:13:10.902323 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.902280 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mttls\" (UniqueName: \"kubernetes.io/projected/8d3ec379-0842-4328-95fe-f2b1f007aabb-kube-api-access-mttls\") pod \"payload-pre-processing-549c5f7d67-lwtfl\" (UID: \"8d3ec379-0842-4328-95fe-f2b1f007aabb\") " pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:10.902497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:10.902474 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3ec379-0842-4328-95fe-f2b1f007aabb-plugins-config-volume\") pod \"payload-pre-processing-549c5f7d67-lwtfl\" (UID: \"8d3ec379-0842-4328-95fe-f2b1f007aabb\") " pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:11.003792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.003712 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3ec379-0842-4328-95fe-f2b1f007aabb-plugins-config-volume\") pod \"payload-pre-processing-549c5f7d67-lwtfl\" (UID: \"8d3ec379-0842-4328-95fe-f2b1f007aabb\") " pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:11.003983 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.003835 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mttls\" (UniqueName: \"kubernetes.io/projected/8d3ec379-0842-4328-95fe-f2b1f007aabb-kube-api-access-mttls\") pod \"payload-pre-processing-549c5f7d67-lwtfl\" (UID: \"8d3ec379-0842-4328-95fe-f2b1f007aabb\") " pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:11.004415 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.004394 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3ec379-0842-4328-95fe-f2b1f007aabb-plugins-config-volume\") pod \"payload-pre-processing-549c5f7d67-lwtfl\" (UID: \"8d3ec379-0842-4328-95fe-f2b1f007aabb\") " pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:11.068150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.068115 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mttls\" (UniqueName: \"kubernetes.io/projected/8d3ec379-0842-4328-95fe-f2b1f007aabb-kube-api-access-mttls\") pod \"payload-pre-processing-549c5f7d67-lwtfl\" (UID: \"8d3ec379-0842-4328-95fe-f2b1f007aabb\") " pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:11.154133 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.154095 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:11.529039 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:13:11.529008 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3ec379_0842_4328_95fe_f2b1f007aabb.slice/crio-b78989d5f5e5a7970132d451e3675455ae529522d4e7a261b1bfbaec9c19c319 WatchSource:0}: Error finding container b78989d5f5e5a7970132d451e3675455ae529522d4e7a261b1bfbaec9c19c319: Status 404 returned error can't find the container with id b78989d5f5e5a7970132d451e3675455ae529522d4e7a261b1bfbaec9c19c319 Nov 01 10:13:11.531442 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.531415 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl"] Nov 01 10:13:11.967330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.967297 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" event={"ID":"8d3ec379-0842-4328-95fe-f2b1f007aabb","Type":"ContainerStarted","Data":"e2035dfe05ce4b22756585070bcaaf16d5d3ade3ca4da85cff35b29f716d9107"} Nov 01 10:13:11.967822 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.967355 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" event={"ID":"8d3ec379-0842-4328-95fe-f2b1f007aabb","Type":"ContainerStarted","Data":"b78989d5f5e5a7970132d451e3675455ae529522d4e7a261b1bfbaec9c19c319"} Nov 01 10:13:11.967822 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:11.967401 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:12.087590 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:12.087534 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" podStartSLOduration=2.087518818 podStartE2EDuration="2.087518818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:13:12.086842514 +0000 UTC m=+6015.175179187" watchObservedRunningTime="2026-11-01 10:13:12.087518818 +0000 UTC m=+6015.175855484" Nov 01 10:13:13.427504 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.427436 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" podUID="1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" containerName="payload-processing" containerID="cri-o://a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824" gracePeriod=25 Nov 01 10:13:13.694528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.694501 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 10:13:13.731074 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.731041 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume\") pod \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " Nov 01 10:13:13.731271 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.731152 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9\" (UniqueName: \"kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9\") pod \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\" (UID: \"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378\") " Nov 01 10:13:13.731457 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.731434 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume" pod "1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" (UID: "1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:13:13.733420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.733398 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9" pod "1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" (UID: "1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378"). InnerVolumeSpecName "kube-api-access-dzqb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:13:13.832292 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.832253 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:13:13.832292 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.832294 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzqb9\" (UniqueName: \"kubernetes.io/projected/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378-kube-api-access-dzqb9\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:13:13.979070 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.978969 5710 generic.go:356] "Generic (PLEG): container finished" podID="1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" containerID="a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824" exitCode=0 Nov 01 10:13:13.979070 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.979023 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" Nov 01 10:13:13.979306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.979060 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" event={"ID":"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378","Type":"ContainerDied","Data":"a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824"} Nov 01 10:13:13.979306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.979144 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-sccmk" event={"ID":"1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378","Type":"ContainerDied","Data":"96087825c53ad653c97e71cd2ef91b0076a19c228abc883557aee0f8d027aa12"} Nov 01 10:13:13.979306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.979169 5710 scope.go:122] "RemoveContainer" containerID="a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824" Nov 01 10:13:13.988661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.988636 5710 scope.go:122] "RemoveContainer" containerID="a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824" Nov 01 10:13:13.988994 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:13:13.988976 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824\": container with ID starting with a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824 not found: ID does not exist" containerID="a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824" Nov 01 10:13:13.989081 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:13.989007 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824"} err="failed to get container status \"a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824\": rpc error: code = NotFound desc = could not find container \"a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824\": container with ID starting with a89dd197ec4643886821b3e85037a8f9931127df7851cb0d1b88586836446824 not found: ID does not exist" Nov 01 10:13:14.151386 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:14.151348 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sccmk"] Nov 01 10:13:14.156563 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:14.156540 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-sccmk"] Nov 01 10:13:15.056796 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:15.056759 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" path="/var/lib/kubelet/pods/1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378/volumes" Nov 01 10:13:15.476913 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:13:15.476877 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:13:22.975140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:22.975100 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-549c5f7d67-lwtfl" Nov 01 10:13:23.124514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:23.124478 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t"] Nov 01 10:13:28.125593 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.125536 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" podUID="9b67adf3-36f7-4990-9406-c163ea5bd45e" containerName="payload-pre-processing" containerID="cri-o://856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c" gracePeriod=25 Nov 01 10:13:28.379453 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.379380 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:13:28.482607 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.482568 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b\" (UniqueName: \"kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b\") pod \"9b67adf3-36f7-4990-9406-c163ea5bd45e\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " Nov 01 10:13:28.482840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.482617 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume\") pod \"9b67adf3-36f7-4990-9406-c163ea5bd45e\" (UID: \"9b67adf3-36f7-4990-9406-c163ea5bd45e\") " Nov 01 10:13:28.483118 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.483082 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume" pod "9b67adf3-36f7-4990-9406-c163ea5bd45e" (UID: "9b67adf3-36f7-4990-9406-c163ea5bd45e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:13:28.485034 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.485005 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b" pod "9b67adf3-36f7-4990-9406-c163ea5bd45e" (UID: "9b67adf3-36f7-4990-9406-c163ea5bd45e"). InnerVolumeSpecName "kube-api-access-6527b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:13:28.584169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.584135 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6527b\" (UniqueName: \"kubernetes.io/projected/9b67adf3-36f7-4990-9406-c163ea5bd45e-kube-api-access-6527b\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:13:28.584169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:28.584165 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b67adf3-36f7-4990-9406-c163ea5bd45e-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:13:29.058047 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.058009 5710 generic.go:356] "Generic (PLEG): container finished" podID="9b67adf3-36f7-4990-9406-c163ea5bd45e" containerID="856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c" exitCode=0 Nov 01 10:13:29.058245 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.058051 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" Nov 01 10:13:29.058245 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.058090 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" event={"ID":"9b67adf3-36f7-4990-9406-c163ea5bd45e","Type":"ContainerDied","Data":"856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c"} Nov 01 10:13:29.058245 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.058148 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t" event={"ID":"9b67adf3-36f7-4990-9406-c163ea5bd45e","Type":"ContainerDied","Data":"df1cb89ba2c55179dece5e2a0783e038b2759e54bb21c3fdcfa871728e2e6cce"} Nov 01 10:13:29.058245 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.058180 5710 scope.go:122] "RemoveContainer" containerID="856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c" Nov 01 10:13:29.068533 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.068509 5710 scope.go:122] "RemoveContainer" containerID="856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c" Nov 01 10:13:29.068863 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:13:29.068838 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c\": container with ID starting with 856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c not found: ID does not exist" containerID="856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c" Nov 01 10:13:29.068967 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.068873 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c"} err="failed to get container status \"856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c\": rpc error: code = NotFound desc = could not find container \"856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c\": container with ID starting with 856c4487936476320d7441bf0b8f602a4d592e11e619ab54829f26cdcbd8688c not found: ID does not exist" Nov 01 10:13:29.228175 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.228126 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t"] Nov 01 10:13:29.279787 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.279750 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-78d99cfb56-mbj9t"] Nov 01 10:13:29.979239 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979194 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs"] Nov 01 10:13:29.979654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979640 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" containerName="payload-processing" Nov 01 10:13:29.979702 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979656 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" containerName="payload-processing" Nov 01 10:13:29.979702 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979680 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9b67adf3-36f7-4990-9406-c163ea5bd45e" containerName="payload-pre-processing" Nov 01 10:13:29.979702 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979686 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b67adf3-36f7-4990-9406-c163ea5bd45e" containerName="payload-pre-processing" Nov 01 10:13:29.979852 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979771 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3dd39c-5a0b-4ad0-a84d-c826fa2d0378" containerName="payload-processing" Nov 01 10:13:29.979852 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.979786 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b67adf3-36f7-4990-9406-c163ea5bd45e" containerName="payload-pre-processing" Nov 01 10:13:29.983955 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:29.983932 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.038742 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.038672 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs"] Nov 01 10:13:30.087406 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.087376 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 01 10:13:30.087634 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.087616 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 10:13:30.087692 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.087660 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 10:13:30.087780 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.087726 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 10:13:30.090266 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.090252 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n4b84\"" Nov 01 10:13:30.090324 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.090253 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 10:13:30.117019 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.116979 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xls\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.117228 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.117031 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.117228 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.117055 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.117353 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.117233 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.218169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.218117 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xls\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.218169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.218188 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.218494 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.218224 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.218494 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.218318 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.219007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.218980 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.219143 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.219048 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.220895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.220872 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.282538 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.282462 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xls\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls\") pod \"payload-processing-external-model-5bcd7997-2tgzs\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.297342 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.297307 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:30.693754 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:13:30.692454 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35912c0_0776_4585_a8f0_1ae542965448.slice/crio-ebb1182a5408876cd87445619712db84abb6409c53b5ee3a8545d6cac6d890b6 WatchSource:0}: Error finding container ebb1182a5408876cd87445619712db84abb6409c53b5ee3a8545d6cac6d890b6: Status 404 returned error can't find the container with id ebb1182a5408876cd87445619712db84abb6409c53b5ee3a8545d6cac6d890b6 Nov 01 10:13:30.693754 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.692515 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs"] Nov 01 10:13:30.852996 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:30.852960 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:13:31.056894 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:31.056817 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b67adf3-36f7-4990-9406-c163ea5bd45e" path="/var/lib/kubelet/pods/9b67adf3-36f7-4990-9406-c163ea5bd45e/volumes" Nov 01 10:13:31.071122 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:31.071090 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" event={"ID":"d35912c0-0776-4585-a8f0-1ae542965448","Type":"ContainerStarted","Data":"ea92634c9ac75c02a8e574b68fef56803ba3c48cd36c2b9c2e36a58c7916c32c"} Nov 01 10:13:31.071312 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:31.071143 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:31.071312 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:31.071161 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" event={"ID":"d35912c0-0776-4585-a8f0-1ae542965448","Type":"ContainerStarted","Data":"ebb1182a5408876cd87445619712db84abb6409c53b5ee3a8545d6cac6d890b6"} Nov 01 10:13:31.214371 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:31.214307 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" podStartSLOduration=2.2142924920000002 podStartE2EDuration="2.214292492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:13:31.212968528 +0000 UTC m=+6034.301305200" watchObservedRunningTime="2026-11-01 10:13:31.214292492 +0000 UTC m=+6034.302629163" Nov 01 10:13:34.899048 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:34.898940 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:13:34.899048 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:34.899023 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:13:42.080521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:13:42.080487 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:13:45.478543 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:13:45.478503 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:14:15.478228 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:14:15.478184 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:14:17.057332 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:14:17.057293 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl"] Nov 01 10:14:17.106689 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:14:17.106653 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172546e2-pjszl"] Nov 01 10:14:19.057533 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:14:19.057493 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80c78da-1674-4453-9021-f37d0d5e7803" path="/var/lib/kubelet/pods/a80c78da-1674-4453-9021-f37d0d5e7803/volumes" Nov 01 10:14:45.476927 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:14:45.476884 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:15:00.291661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.291616 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs"] Nov 01 10:15:00.295671 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.295632 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs"] Nov 01 10:15:00.295871 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.295787 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.396232 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.396193 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7r66\" (UniqueName: \"kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66\") pod \"maas-api-key-cleanup-29892135-d77bs\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.396438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.396314 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892135-d77bs\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.402490 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.402461 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:15:00.402490 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.402498 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:15:00.402804 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.402525 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:15:00.497386 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.497343 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7r66\" (UniqueName: \"kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66\") pod \"maas-api-key-cleanup-29892135-d77bs\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.497602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.497432 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892135-d77bs\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.498175 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.498148 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892135-d77bs\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.615887 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.615850 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7r66\" (UniqueName: \"kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66\") pod \"maas-api-key-cleanup-29892135-d77bs\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:00.909535 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:00.909457 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:01.282521 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:15:01.282476 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad54a72b_b492_4dfe_b47a_93a4dc15a181.slice/crio-408858314c214836aebf08884ed23fa58ead796040639f0ab9b3ae3b8db82f16 WatchSource:0}: Error finding container 408858314c214836aebf08884ed23fa58ead796040639f0ab9b3ae3b8db82f16: Status 404 returned error can't find the container with id 408858314c214836aebf08884ed23fa58ead796040639f0ab9b3ae3b8db82f16 Nov 01 10:15:01.284451 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:01.284419 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs"] Nov 01 10:15:01.590009 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:01.589970 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" event={"ID":"ad54a72b-b492-4dfe-b47a-93a4dc15a181","Type":"ContainerStarted","Data":"408858314c214836aebf08884ed23fa58ead796040639f0ab9b3ae3b8db82f16"} Nov 01 10:15:02.597855 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:02.597822 5710 generic.go:356] "Generic (PLEG): container finished" podID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerID="5ba2bdadd5ca4bf51731cee2744669db8134db520be45349f326c6c5866021a3" exitCode=60 Nov 01 10:15:02.598283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:02.597905 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" event={"ID":"ad54a72b-b492-4dfe-b47a-93a4dc15a181","Type":"ContainerDied","Data":"5ba2bdadd5ca4bf51731cee2744669db8134db520be45349f326c6c5866021a3"} Nov 01 10:15:02.598283 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:02.598180 5710 scope.go:122] "RemoveContainer" containerID="5ba2bdadd5ca4bf51731cee2744669db8134db520be45349f326c6c5866021a3" Nov 01 10:15:03.605630 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:03.605596 5710 generic.go:356] "Generic (PLEG): container finished" podID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerID="8b5ca62ca47f8a3d1e7f48b9b34f1974e2c23184718fda562d6f6358aa4c12a5" exitCode=60 Nov 01 10:15:03.606164 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:03.605693 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" event={"ID":"ad54a72b-b492-4dfe-b47a-93a4dc15a181","Type":"ContainerDied","Data":"8b5ca62ca47f8a3d1e7f48b9b34f1974e2c23184718fda562d6f6358aa4c12a5"} Nov 01 10:15:03.606164 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:03.605802 5710 scope.go:122] "RemoveContainer" containerID="5ba2bdadd5ca4bf51731cee2744669db8134db520be45349f326c6c5866021a3" Nov 01 10:15:03.606164 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:03.605999 5710 scope.go:122] "RemoveContainer" containerID="8b5ca62ca47f8a3d1e7f48b9b34f1974e2c23184718fda562d6f6358aa4c12a5" Nov 01 10:15:03.606328 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:15:03.606303 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892135-d77bs_odh-ai-gateway-infra(ad54a72b-b492-4dfe-b47a-93a4dc15a181)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" Nov 01 10:15:04.612534 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:04.612460 5710 scope.go:122] "RemoveContainer" containerID="8b5ca62ca47f8a3d1e7f48b9b34f1974e2c23184718fda562d6f6358aa4c12a5" Nov 01 10:15:04.612931 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:15:04.612763 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892135-d77bs_odh-ai-gateway-infra(ad54a72b-b492-4dfe-b47a-93a4dc15a181)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" Nov 01 10:15:15.051083 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:15.051044 5710 scope.go:122] "RemoveContainer" containerID="8b5ca62ca47f8a3d1e7f48b9b34f1974e2c23184718fda562d6f6358aa4c12a5" Nov 01 10:15:15.478296 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:15:15.478253 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:15:15.672105 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:15.672011 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" event={"ID":"ad54a72b-b492-4dfe-b47a-93a4dc15a181","Type":"ContainerStarted","Data":"e81868a135391a507721500218932031c7528e4ef130cdcb8cb0238d31df6bee"} Nov 01 10:15:15.845061 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:15.845004 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" podStartSLOduration=15.844990163 podStartE2EDuration="15.844990163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:15:15.843568851 +0000 UTC m=+6138.931905524" watchObservedRunningTime="2026-11-01 10:15:15.844990163 +0000 UTC m=+6138.933326829" Nov 01 10:15:16.684782 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:16.684738 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" event={"ID":"ad54a72b-b492-4dfe-b47a-93a4dc15a181","Type":"ContainerDied","Data":"e81868a135391a507721500218932031c7528e4ef130cdcb8cb0238d31df6bee"} Nov 01 10:15:16.684782 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:16.684692 5710 generic.go:356] "Generic (PLEG): container finished" podID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerID="e81868a135391a507721500218932031c7528e4ef130cdcb8cb0238d31df6bee" exitCode=60 Nov 01 10:15:16.685397 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:16.685217 5710 scope.go:122] "RemoveContainer" containerID="8b5ca62ca47f8a3d1e7f48b9b34f1974e2c23184718fda562d6f6358aa4c12a5" Nov 01 10:15:16.685673 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:16.685648 5710 scope.go:122] "RemoveContainer" containerID="e81868a135391a507721500218932031c7528e4ef130cdcb8cb0238d31df6bee" Nov 01 10:15:16.686183 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:15:16.686152 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892135-d77bs_odh-ai-gateway-infra(ad54a72b-b492-4dfe-b47a-93a4dc15a181)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" Nov 01 10:15:16.802438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:16.802399 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs"] Nov 01 10:15:17.837954 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.837926 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:17.886298 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.886262 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca\") pod \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " Nov 01 10:15:17.886485 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.886395 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66\" (UniqueName: \"kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66\") pod \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\" (UID: \"ad54a72b-b492-4dfe-b47a-93a4dc15a181\") " Nov 01 10:15:17.886674 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.886649 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca" pod "ad54a72b-b492-4dfe-b47a-93a4dc15a181" (UID: "ad54a72b-b492-4dfe-b47a-93a4dc15a181"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:15:17.888784 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.888759 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66" pod "ad54a72b-b492-4dfe-b47a-93a4dc15a181" (UID: "ad54a72b-b492-4dfe-b47a-93a4dc15a181"). InnerVolumeSpecName "kube-api-access-k7r66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:15:17.987511 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.987422 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7r66\" (UniqueName: \"kubernetes.io/projected/ad54a72b-b492-4dfe-b47a-93a4dc15a181-kube-api-access-k7r66\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:17.987511 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:17.987455 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad54a72b-b492-4dfe-b47a-93a4dc15a181-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:18.700981 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:18.700946 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" Nov 01 10:15:18.701212 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:18.700983 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs" event={"ID":"ad54a72b-b492-4dfe-b47a-93a4dc15a181","Type":"ContainerDied","Data":"408858314c214836aebf08884ed23fa58ead796040639f0ab9b3ae3b8db82f16"} Nov 01 10:15:18.701212 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:18.701047 5710 scope.go:122] "RemoveContainer" containerID="e81868a135391a507721500218932031c7528e4ef130cdcb8cb0238d31df6bee" Nov 01 10:15:18.871540 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:18.871501 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs"] Nov 01 10:15:18.923579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:18.923543 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892135-d77bs"] Nov 01 10:15:19.011743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.011635 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn"] Nov 01 10:15:19.012078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012066 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012080 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012102 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012107 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012117 5710 cpu_manager.go:414] "containerMap: removing container" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012123 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012290 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012212 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.012290 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.012222 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:19.016530 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.016513 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.057374 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.057342 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" path="/var/lib/kubelet/pods/ad54a72b-b492-4dfe-b47a-93a4dc15a181/volumes" Nov 01 10:15:19.070567 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.070528 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn"] Nov 01 10:15:19.100090 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.100055 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6c6t\" (UniqueName: \"kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.100279 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.100108 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.100279 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.100224 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.120269 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.120240 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:15:19.120269 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.120240 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:15:19.120269 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.120250 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:15:19.122926 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.122902 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:15:19.201080 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.201040 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.201295 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.201142 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6c6t\" (UniqueName: \"kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.201295 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.201170 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.201800 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.201702 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.203834 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.203808 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.265260 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.265188 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6c6t\" (UniqueName: \"kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t\") pod \"maas-api-revoke-sub-e2547622-chlbn\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.329977 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.329935 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:19.704502 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:15:19.704462 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfc4516_bdd1_433a_a4ca_7d1ab5779dcf.slice/crio-44716d9a79b3bb88c36b79907919fa4cec3285e02f6cc7f92b0815022ff64149 WatchSource:0}: Error finding container 44716d9a79b3bb88c36b79907919fa4cec3285e02f6cc7f92b0815022ff64149: Status 404 returned error can't find the container with id 44716d9a79b3bb88c36b79907919fa4cec3285e02f6cc7f92b0815022ff64149 Nov 01 10:15:19.706661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:19.706633 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn"] Nov 01 10:15:20.718116 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:20.718080 5710 generic.go:356] "Generic (PLEG): container finished" podID="cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" containerID="815a3e99afb9e5a5916657631f3b0706e5486fb506863b9fad235fe4be34f1f9" exitCode=0 Nov 01 10:15:20.718532 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:20.718168 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" event={"ID":"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf","Type":"ContainerDied","Data":"815a3e99afb9e5a5916657631f3b0706e5486fb506863b9fad235fe4be34f1f9"} Nov 01 10:15:20.718532 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:20.718230 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" event={"ID":"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf","Type":"ContainerStarted","Data":"44716d9a79b3bb88c36b79907919fa4cec3285e02f6cc7f92b0815022ff64149"} Nov 01 10:15:21.872221 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.872194 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:21.930308 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.930267 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca\") pod \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " Nov 01 10:15:21.930542 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.930362 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca\") pod \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " Nov 01 10:15:21.930542 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.930414 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t\" (UniqueName: \"kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t\") pod \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\" (UID: \"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf\") " Nov 01 10:15:21.930709 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.930684 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca" pod "cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" (UID: "cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:15:21.932853 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.932820 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca" pod "cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" (UID: "cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:15:21.932970 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:21.932854 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t" pod "cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" (UID: "cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf"). InnerVolumeSpecName "kube-api-access-m6c6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:15:22.031978 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:22.031880 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:22.031978 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:22.031913 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:22.031978 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:22.031924 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6c6t\" (UniqueName: \"kubernetes.io/projected/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf-kube-api-access-m6c6t\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:22.731649 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:22.731606 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" event={"ID":"cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf","Type":"ContainerDied","Data":"44716d9a79b3bb88c36b79907919fa4cec3285e02f6cc7f92b0815022ff64149"} Nov 01 10:15:22.731895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:22.731690 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44716d9a79b3bb88c36b79907919fa4cec3285e02f6cc7f92b0815022ff64149" Nov 01 10:15:22.731895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:22.731619 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn" Nov 01 10:15:30.480054 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:30.480020 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs"] Nov 01 10:15:33.495492 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:33.495452 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:15:35.480556 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:35.480503 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" podUID="d35912c0-0776-4585-a8f0-1ae542965448" containerName="payload-processing" containerID="cri-o://ea92634c9ac75c02a8e574b68fef56803ba3c48cd36c2b9c2e36a58c7916c32c" gracePeriod=25 Nov 01 10:15:35.874654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:35.874520 5710 generic.go:356] "Generic (PLEG): container finished" podID="d35912c0-0776-4585-a8f0-1ae542965448" containerID="ea92634c9ac75c02a8e574b68fef56803ba3c48cd36c2b9c2e36a58c7916c32c" exitCode=0 Nov 01 10:15:35.874654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:35.874583 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" event={"ID":"d35912c0-0776-4585-a8f0-1ae542965448","Type":"ContainerDied","Data":"ea92634c9ac75c02a8e574b68fef56803ba3c48cd36c2b9c2e36a58c7916c32c"} Nov 01 10:15:35.990746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:35.981766 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:15:36.074554 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.074447 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume\") pod \"d35912c0-0776-4585-a8f0-1ae542965448\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " Nov 01 10:15:36.074554 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.074507 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume\") pod \"d35912c0-0776-4585-a8f0-1ae542965448\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " Nov 01 10:15:36.074856 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.074585 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials\") pod \"d35912c0-0776-4585-a8f0-1ae542965448\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " Nov 01 10:15:36.074856 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.074675 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls\") pod \"d35912c0-0776-4585-a8f0-1ae542965448\" (UID: \"d35912c0-0776-4585-a8f0-1ae542965448\") " Nov 01 10:15:36.074979 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.074927 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume" pod "d35912c0-0776-4585-a8f0-1ae542965448" (UID: "d35912c0-0776-4585-a8f0-1ae542965448"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:15:36.075041 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.074987 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume" pod "d35912c0-0776-4585-a8f0-1ae542965448" (UID: "d35912c0-0776-4585-a8f0-1ae542965448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:15:36.075185 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.075170 5710 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-routing-overlay-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:36.075252 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.075188 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35912c0-0776-4585-a8f0-1ae542965448-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:36.077185 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.077152 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls" pod "d35912c0-0776-4585-a8f0-1ae542965448" (UID: "d35912c0-0776-4585-a8f0-1ae542965448"). InnerVolumeSpecName "kube-api-access-h9xls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:15:36.077185 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.077159 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials" pod "d35912c0-0776-4585-a8f0-1ae542965448" (UID: "d35912c0-0776-4585-a8f0-1ae542965448"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:15:36.175910 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.175857 5710 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-provider-credentials\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:36.175910 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.175896 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xls\" (UniqueName: \"kubernetes.io/projected/d35912c0-0776-4585-a8f0-1ae542965448-kube-api-access-h9xls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:36.881261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.881234 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" Nov 01 10:15:36.881840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.881224 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs" event={"ID":"d35912c0-0776-4585-a8f0-1ae542965448","Type":"ContainerDied","Data":"ebb1182a5408876cd87445619712db84abb6409c53b5ee3a8545d6cac6d890b6"} Nov 01 10:15:36.881840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:36.881377 5710 scope.go:122] "RemoveContainer" containerID="ea92634c9ac75c02a8e574b68fef56803ba3c48cd36c2b9c2e36a58c7916c32c" Nov 01 10:15:37.057699 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:37.057665 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs"] Nov 01 10:15:37.105128 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:37.105088 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-2tgzs"] Nov 01 10:15:39.057643 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:39.057609 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35912c0-0776-4585-a8f0-1ae542965448" path="/var/lib/kubelet/pods/d35912c0-0776-4585-a8f0-1ae542965448/volumes" Nov 01 10:15:43.857450 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:43.857390 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:15:45.496744 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:15:45.495935 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:15:50.487432 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.487383 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk"] Nov 01 10:15:50.488068 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488044 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" containerName="revoke-keys" Nov 01 10:15:50.488170 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488071 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" containerName="revoke-keys" Nov 01 10:15:50.488170 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488125 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d35912c0-0776-4585-a8f0-1ae542965448" containerName="payload-processing" Nov 01 10:15:50.488170 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488133 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35912c0-0776-4585-a8f0-1ae542965448" containerName="payload-processing" Nov 01 10:15:50.488330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488247 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35912c0-0776-4585-a8f0-1ae542965448" containerName="payload-processing" Nov 01 10:15:50.488330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488259 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad54a72b-b492-4dfe-b47a-93a4dc15a181" containerName="cleanup" Nov 01 10:15:50.488330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.488274 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" containerName="revoke-keys" Nov 01 10:15:50.491532 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.491507 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.547760 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.547693 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk"] Nov 01 10:15:50.564918 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.564882 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfx8\" (UniqueName: \"kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.564918 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.564929 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.565158 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.565059 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.597443 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.597403 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:15:50.597639 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.597453 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:15:50.597639 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.597482 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:15:50.597994 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.597973 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:15:50.666276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.666227 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfx8\" (UniqueName: \"kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.666521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.666284 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.666521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.666353 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.667180 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.667153 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.669120 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.669095 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.730000 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.729970 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfx8\" (UniqueName: \"kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8\") pod \"maas-api-revoke-sub-c1e271a9-tzrxk\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:50.808325 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:50.808223 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:51.181380 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:15:51.181333 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386bcb96_8863_4cae_9458_3a33f5969612.slice/crio-a4031f099883a57f63f3be360f43664dafadb11a0a3c5af74b0008b4ecd31eb9 WatchSource:0}: Error finding container a4031f099883a57f63f3be360f43664dafadb11a0a3c5af74b0008b4ecd31eb9: Status 404 returned error can't find the container with id a4031f099883a57f63f3be360f43664dafadb11a0a3c5af74b0008b4ecd31eb9 Nov 01 10:15:51.183266 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:51.183186 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk"] Nov 01 10:15:51.991091 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:51.991051 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" event={"ID":"386bcb96-8863-4cae-9458-3a33f5969612","Type":"ContainerStarted","Data":"7ce0c3fa5b0bf4765e31354189857d141c1f5d608a2c31434e6a5e2698470aee"} Nov 01 10:15:51.991531 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:51.991118 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" event={"ID":"386bcb96-8863-4cae-9458-3a33f5969612","Type":"ContainerStarted","Data":"a4031f099883a57f63f3be360f43664dafadb11a0a3c5af74b0008b4ecd31eb9"} Nov 01 10:15:52.116059 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:52.116002 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" podStartSLOduration=2.115987367 podStartE2EDuration="2.115987367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:15:52.114992389 +0000 UTC m=+6175.203329060" watchObservedRunningTime="2026-11-01 10:15:52.115987367 +0000 UTC m=+6175.204324039" Nov 01 10:15:52.997433 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:52.997392 5710 generic.go:356] "Generic (PLEG): container finished" podID="386bcb96-8863-4cae-9458-3a33f5969612" containerID="7ce0c3fa5b0bf4765e31354189857d141c1f5d608a2c31434e6a5e2698470aee" exitCode=0 Nov 01 10:15:52.997956 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:52.997482 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" event={"ID":"386bcb96-8863-4cae-9458-3a33f5969612","Type":"ContainerDied","Data":"7ce0c3fa5b0bf4765e31354189857d141c1f5d608a2c31434e6a5e2698470aee"} Nov 01 10:15:54.157190 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.157153 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:54.303752 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.303631 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca\") pod \"386bcb96-8863-4cae-9458-3a33f5969612\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " Nov 01 10:15:54.303752 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.303708 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca\") pod \"386bcb96-8863-4cae-9458-3a33f5969612\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " Nov 01 10:15:54.304032 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.303795 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8\" (UniqueName: \"kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8\") pod \"386bcb96-8863-4cae-9458-3a33f5969612\" (UID: \"386bcb96-8863-4cae-9458-3a33f5969612\") " Nov 01 10:15:54.304132 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.304104 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca" pod "386bcb96-8863-4cae-9458-3a33f5969612" (UID: "386bcb96-8863-4cae-9458-3a33f5969612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:15:54.304267 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.304188 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:54.306069 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.306040 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca" pod "386bcb96-8863-4cae-9458-3a33f5969612" (UID: "386bcb96-8863-4cae-9458-3a33f5969612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:15:54.306262 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.306237 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8" pod "386bcb96-8863-4cae-9458-3a33f5969612" (UID: "386bcb96-8863-4cae-9458-3a33f5969612"). InnerVolumeSpecName "kube-api-access-dtfx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:15:54.405120 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.405080 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bcb96-8863-4cae-9458-3a33f5969612-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:54.405120 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:54.405116 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfx8\" (UniqueName: \"kubernetes.io/projected/386bcb96-8863-4cae-9458-3a33f5969612-kube-api-access-dtfx8\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:15:55.008264 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:55.008223 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" event={"ID":"386bcb96-8863-4cae-9458-3a33f5969612","Type":"ContainerDied","Data":"a4031f099883a57f63f3be360f43664dafadb11a0a3c5af74b0008b4ecd31eb9"} Nov 01 10:15:55.008264 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:55.008248 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk" Nov 01 10:15:55.008521 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:15:55.008302 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4031f099883a57f63f3be360f43664dafadb11a0a3c5af74b0008b4ecd31eb9" Nov 01 10:16:01.262955 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:01.262920 5710 scope.go:122] "RemoveContainer" containerID="4ff4f7be5d84187e30c220149979c7eac0bdd58d042c85704748ccaac7d4d43f" Nov 01 10:16:03.066756 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:03.066698 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:16:12.668881 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:12.668836 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:16:12.669388 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:12.668899 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:16:13.603683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:13.603644 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:16:14.927613 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:14.927578 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794"] Nov 01 10:16:14.928085 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:14.928054 5710 cpu_manager.go:414] "containerMap: removing container" podUID="386bcb96-8863-4cae-9458-3a33f5969612" containerName="revoke-keys" Nov 01 10:16:14.928085 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:14.928068 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386bcb96-8863-4cae-9458-3a33f5969612" containerName="revoke-keys" Nov 01 10:16:14.928181 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:14.928158 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="386bcb96-8863-4cae-9458-3a33f5969612" containerName="revoke-keys" Nov 01 10:16:14.931482 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:14.931460 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:14.986648 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:14.986612 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794"] Nov 01 10:16:15.036069 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.036030 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:16:15.036069 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.036050 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:16:15.036069 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.036055 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:16:15.037398 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.037381 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:16:15.060044 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.060013 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.060242 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.060052 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4mt\" (UniqueName: \"kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.060242 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.060210 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.161491 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.161456 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.161698 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.161527 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.161698 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.161566 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4mt\" (UniqueName: \"kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.162243 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.162219 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.164041 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.164022 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.225112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.225040 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4mt\" (UniqueName: \"kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt\") pod \"maas-api-revoke-sub-2e1f5b5f-8f794\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.249929 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.249888 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:15.477886 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:16:15.477781 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:16:15.627827 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:16:15.627787 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3e85ef_7cd2_41a7_987a_fbc1ae4bbef3.slice/crio-e6ed334bb86e40e8ab6bde32e2d02b07777ea34abf421454439cef4c8e8725b1 WatchSource:0}: Error finding container e6ed334bb86e40e8ab6bde32e2d02b07777ea34abf421454439cef4c8e8725b1: Status 404 returned error can't find the container with id e6ed334bb86e40e8ab6bde32e2d02b07777ea34abf421454439cef4c8e8725b1 Nov 01 10:16:15.630049 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:15.630022 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794"] Nov 01 10:16:16.149350 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:16.149311 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" event={"ID":"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3","Type":"ContainerStarted","Data":"7101f97abfc02595dcb44c6a0e672968adaf00b7f81e297cd5c0c5aaab17e92c"} Nov 01 10:16:16.149956 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:16.149363 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" event={"ID":"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3","Type":"ContainerStarted","Data":"e6ed334bb86e40e8ab6bde32e2d02b07777ea34abf421454439cef4c8e8725b1"} Nov 01 10:16:16.269199 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:16.269141 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" podStartSLOduration=2.269120839 podStartE2EDuration="2.269120839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:16:16.268846891 +0000 UTC m=+6199.357183563" watchObservedRunningTime="2026-11-01 10:16:16.269120839 +0000 UTC m=+6199.357457510" Nov 01 10:16:17.156347 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:17.156308 5710 generic.go:356] "Generic (PLEG): container finished" podID="3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" containerID="7101f97abfc02595dcb44c6a0e672968adaf00b7f81e297cd5c0c5aaab17e92c" exitCode=0 Nov 01 10:16:17.156868 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:17.156389 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" event={"ID":"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3","Type":"ContainerDied","Data":"7101f97abfc02595dcb44c6a0e672968adaf00b7f81e297cd5c0c5aaab17e92c"} Nov 01 10:16:18.328886 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.328855 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:18.497132 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.497036 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca\") pod \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " Nov 01 10:16:18.497132 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.497129 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt\" (UniqueName: \"kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt\") pod \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " Nov 01 10:16:18.497361 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.497229 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca\") pod \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\" (UID: \"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3\") " Nov 01 10:16:18.497501 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.497481 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca" pod "3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" (UID: "3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:16:18.497616 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.497596 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:16:18.499633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.499599 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt" pod "3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" (UID: "3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3"). InnerVolumeSpecName "kube-api-access-4l4mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:16:18.499633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.499595 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca" pod "3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" (UID: "3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:16:18.598574 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.598530 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4mt\" (UniqueName: \"kubernetes.io/projected/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-kube-api-access-4l4mt\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:16:18.598574 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:18.598566 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:16:19.170613 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:19.170573 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" event={"ID":"3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3","Type":"ContainerDied","Data":"e6ed334bb86e40e8ab6bde32e2d02b07777ea34abf421454439cef4c8e8725b1"} Nov 01 10:16:19.170854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:19.170629 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ed334bb86e40e8ab6bde32e2d02b07777ea34abf421454439cef4c8e8725b1" Nov 01 10:16:19.170854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:19.170584 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794" Nov 01 10:16:27.214053 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.214019 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf"] Nov 01 10:16:27.214470 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.214444 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" containerName="revoke-keys" Nov 01 10:16:27.214470 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.214455 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" containerName="revoke-keys" Nov 01 10:16:27.214547 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.214538 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" containerName="revoke-keys" Nov 01 10:16:27.217828 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.217808 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.273672 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.273635 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf"] Nov 01 10:16:27.283418 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.283376 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.283602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.283437 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb2m\" (UniqueName: \"kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.283602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.283534 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.322648 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.322619 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:16:27.322648 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.322628 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:16:27.322876 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.322672 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:16:27.323761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.323736 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:16:27.384936 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.384898 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.385160 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.384959 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb2m\" (UniqueName: \"kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.385160 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.384987 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.385634 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.385607 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.387577 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.387548 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.448862 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.448824 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb2m\" (UniqueName: \"kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m\") pod \"maas-api-revoke-sub-ec843018-4h9pf\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.531342 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.531250 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:27.906924 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:16:27.906885 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1161c0a3_231b_48c8_a2f4_8ffc26882f2b.slice/crio-d1e6649005a8345044f6d3639e4da3ba24883685c9732065c386937edaa7ddb5 WatchSource:0}: Error finding container d1e6649005a8345044f6d3639e4da3ba24883685c9732065c386937edaa7ddb5: Status 404 returned error can't find the container with id d1e6649005a8345044f6d3639e4da3ba24883685c9732065c386937edaa7ddb5 Nov 01 10:16:27.908598 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:27.908576 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf"] Nov 01 10:16:28.220164 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:28.220113 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" event={"ID":"1161c0a3-231b-48c8-a2f4-8ffc26882f2b","Type":"ContainerStarted","Data":"2e29f4927a436d6a3687e232618184592355825bfb1b80276661dbb7a3307ac7"} Nov 01 10:16:28.220645 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:28.220193 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" event={"ID":"1161c0a3-231b-48c8-a2f4-8ffc26882f2b","Type":"ContainerStarted","Data":"d1e6649005a8345044f6d3639e4da3ba24883685c9732065c386937edaa7ddb5"} Nov 01 10:16:28.340255 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:28.340199 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" podStartSLOduration=1.340184308 podStartE2EDuration="1.340184308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:16:28.339920044 +0000 UTC m=+6211.428256717" watchObservedRunningTime="2026-11-01 10:16:28.340184308 +0000 UTC m=+6211.428520979" Nov 01 10:16:29.226581 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:29.226546 5710 generic.go:356] "Generic (PLEG): container finished" podID="1161c0a3-231b-48c8-a2f4-8ffc26882f2b" containerID="2e29f4927a436d6a3687e232618184592355825bfb1b80276661dbb7a3307ac7" exitCode=0 Nov 01 10:16:29.227073 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:29.226637 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" event={"ID":"1161c0a3-231b-48c8-a2f4-8ffc26882f2b","Type":"ContainerDied","Data":"2e29f4927a436d6a3687e232618184592355825bfb1b80276661dbb7a3307ac7"} Nov 01 10:16:30.380742 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.380698 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:30.419891 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.419851 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca\") pod \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " Nov 01 10:16:30.420120 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.419950 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca\") pod \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " Nov 01 10:16:30.420120 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.420039 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m\" (UniqueName: \"kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m\") pod \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\" (UID: \"1161c0a3-231b-48c8-a2f4-8ffc26882f2b\") " Nov 01 10:16:30.420286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.420259 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca" pod "1161c0a3-231b-48c8-a2f4-8ffc26882f2b" (UID: "1161c0a3-231b-48c8-a2f4-8ffc26882f2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:16:30.422497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.422463 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca" pod "1161c0a3-231b-48c8-a2f4-8ffc26882f2b" (UID: "1161c0a3-231b-48c8-a2f4-8ffc26882f2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:16:30.422640 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.422562 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m" pod "1161c0a3-231b-48c8-a2f4-8ffc26882f2b" (UID: "1161c0a3-231b-48c8-a2f4-8ffc26882f2b"). InnerVolumeSpecName "kube-api-access-gpb2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:16:30.520896 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.520796 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpb2m\" (UniqueName: \"kubernetes.io/projected/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-kube-api-access-gpb2m\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:16:30.520896 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.520832 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:16:30.520896 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.520844 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1161c0a3-231b-48c8-a2f4-8ffc26882f2b-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:16:30.978246 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:30.978210 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:16:31.238302 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:31.238219 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" Nov 01 10:16:31.238302 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:31.238252 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf" event={"ID":"1161c0a3-231b-48c8-a2f4-8ffc26882f2b","Type":"ContainerDied","Data":"d1e6649005a8345044f6d3639e4da3ba24883685c9732065c386937edaa7ddb5"} Nov 01 10:16:31.238302 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:31.238295 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e6649005a8345044f6d3639e4da3ba24883685c9732065c386937edaa7ddb5" Nov 01 10:16:44.287861 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:16:44.287757 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:16:45.486299 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:16:45.486262 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:17:02.776351 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.776315 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh"] Nov 01 10:17:02.776905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.776886 5710 cpu_manager.go:414] "containerMap: removing container" podUID="1161c0a3-231b-48c8-a2f4-8ffc26882f2b" containerName="revoke-keys" Nov 01 10:17:02.776905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.776906 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1161c0a3-231b-48c8-a2f4-8ffc26882f2b" containerName="revoke-keys" Nov 01 10:17:02.777066 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.776976 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="1161c0a3-231b-48c8-a2f4-8ffc26882f2b" containerName="revoke-keys" Nov 01 10:17:02.780367 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.780348 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.824558 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.824515 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.824841 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.824576 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wpf\" (UniqueName: \"kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.824923 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.824844 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.834539 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.834496 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh"] Nov 01 10:17:02.884510 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.884473 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:17:02.886526 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.886505 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:17:02.886638 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.886534 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:17:02.886638 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.886542 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:17:02.925783 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.925740 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.926003 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.925801 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.926003 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.925847 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wpf\" (UniqueName: \"kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.926424 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.926402 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.928542 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.928514 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:02.989985 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:02.989951 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wpf\" (UniqueName: \"kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf\") pod \"maas-api-revoke-sub-046e1343-jtzmh\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:03.093871 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:03.093828 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:03.473469 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:17:03.473434 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e56962a_0a15_4650_99dd_a3bceeee3ae6.slice/crio-1b6e578ff448b4120b58e17541f453edad4ea846e3a627b7bbcb58b664fe6ef8 WatchSource:0}: Error finding container 1b6e578ff448b4120b58e17541f453edad4ea846e3a627b7bbcb58b664fe6ef8: Status 404 returned error can't find the container with id 1b6e578ff448b4120b58e17541f453edad4ea846e3a627b7bbcb58b664fe6ef8 Nov 01 10:17:03.475041 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:03.475005 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh"] Nov 01 10:17:04.449900 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:04.449859 5710 generic.go:356] "Generic (PLEG): container finished" podID="0e56962a-0a15-4650-99dd-a3bceeee3ae6" containerID="e46ca2732543f2da77db2e8c8ae780c552b9f25cd5db0148631093757caf37a6" exitCode=0 Nov 01 10:17:04.450339 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:04.449921 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" event={"ID":"0e56962a-0a15-4650-99dd-a3bceeee3ae6","Type":"ContainerDied","Data":"e46ca2732543f2da77db2e8c8ae780c552b9f25cd5db0148631093757caf37a6"} Nov 01 10:17:04.450339 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:04.449985 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" event={"ID":"0e56962a-0a15-4650-99dd-a3bceeee3ae6","Type":"ContainerStarted","Data":"1b6e578ff448b4120b58e17541f453edad4ea846e3a627b7bbcb58b664fe6ef8"} Nov 01 10:17:05.599115 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.599078 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:05.655446 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.655411 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca\") pod \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " Nov 01 10:17:05.655658 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.655508 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca\") pod \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " Nov 01 10:17:05.655658 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.655655 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf\" (UniqueName: \"kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf\") pod \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\" (UID: \"0e56962a-0a15-4650-99dd-a3bceeee3ae6\") " Nov 01 10:17:05.655939 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.655916 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca" pod "0e56962a-0a15-4650-99dd-a3bceeee3ae6" (UID: "0e56962a-0a15-4650-99dd-a3bceeee3ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:17:05.656055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.656038 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:05.657897 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.657872 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca" pod "0e56962a-0a15-4650-99dd-a3bceeee3ae6" (UID: "0e56962a-0a15-4650-99dd-a3bceeee3ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:17:05.658001 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.657964 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf" pod "0e56962a-0a15-4650-99dd-a3bceeee3ae6" (UID: "0e56962a-0a15-4650-99dd-a3bceeee3ae6"). InnerVolumeSpecName "kube-api-access-v8wpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:17:05.757397 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.757314 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wpf\" (UniqueName: \"kubernetes.io/projected/0e56962a-0a15-4650-99dd-a3bceeee3ae6-kube-api-access-v8wpf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:05.757397 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:05.757344 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e56962a-0a15-4650-99dd-a3bceeee3ae6-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:06.462056 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:06.462025 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" Nov 01 10:17:06.462253 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:06.462057 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh" event={"ID":"0e56962a-0a15-4650-99dd-a3bceeee3ae6","Type":"ContainerDied","Data":"1b6e578ff448b4120b58e17541f453edad4ea846e3a627b7bbcb58b664fe6ef8"} Nov 01 10:17:06.462253 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:06.462107 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b6e578ff448b4120b58e17541f453edad4ea846e3a627b7bbcb58b664fe6ef8" Nov 01 10:17:13.507786 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.507749 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj"] Nov 01 10:17:13.508352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.508225 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0e56962a-0a15-4650-99dd-a3bceeee3ae6" containerName="revoke-keys" Nov 01 10:17:13.508352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.508237 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e56962a-0a15-4650-99dd-a3bceeee3ae6" containerName="revoke-keys" Nov 01 10:17:13.508352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.508311 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e56962a-0a15-4650-99dd-a3bceeee3ae6" containerName="revoke-keys" Nov 01 10:17:13.512226 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.512199 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.566631 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.566598 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:17:13.618538 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.618496 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:17:13.618538 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.618507 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:17:13.618881 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.618535 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:17:13.620540 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.620509 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj"] Nov 01 10:17:13.643913 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.643882 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.644105 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.643928 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvgw\" (UniqueName: \"kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.644105 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.644002 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.744701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.744656 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.744960 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.744764 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvgw\" (UniqueName: \"kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.744960 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.744799 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.745444 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.745422 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.747395 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.747371 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.808889 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.808802 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvgw\" (UniqueName: \"kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw\") pod \"maas-api-revoke-sub-d3fba056-zljsj\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:13.825650 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:13.825612 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:14.206138 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:17:14.206094 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1c9053_af28_41e3_b608_69ce5329e3ad.slice/crio-18c8d7702309846e820d16d104c2bbd05756d3cdcbdb4328213a23ecf66a035f WatchSource:0}: Error finding container 18c8d7702309846e820d16d104c2bbd05756d3cdcbdb4328213a23ecf66a035f: Status 404 returned error can't find the container with id 18c8d7702309846e820d16d104c2bbd05756d3cdcbdb4328213a23ecf66a035f Nov 01 10:17:14.207782 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:14.207685 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj"] Nov 01 10:17:14.502624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:14.502582 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" event={"ID":"9c1c9053-af28-41e3-b608-69ce5329e3ad","Type":"ContainerStarted","Data":"4f78e6de7554abea27ce249c5a69b1c4861ead222cf96d11375679fa0ac9eb5c"} Nov 01 10:17:14.502817 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:14.502686 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" event={"ID":"9c1c9053-af28-41e3-b608-69ce5329e3ad","Type":"ContainerStarted","Data":"18c8d7702309846e820d16d104c2bbd05756d3cdcbdb4328213a23ecf66a035f"} Nov 01 10:17:14.622905 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:14.622841 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" podStartSLOduration=1.6228263200000002 podStartE2EDuration="1.62282632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:17:14.622304337 +0000 UTC m=+6257.710641008" watchObservedRunningTime="2026-11-01 10:17:14.62282632 +0000 UTC m=+6257.711162993" Nov 01 10:17:15.478360 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:17:15.478324 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:17:15.484958 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:15.484918 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:17:15.509823 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:15.509781 5710 generic.go:356] "Generic (PLEG): container finished" podID="9c1c9053-af28-41e3-b608-69ce5329e3ad" containerID="4f78e6de7554abea27ce249c5a69b1c4861ead222cf96d11375679fa0ac9eb5c" exitCode=0 Nov 01 10:17:15.510057 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:15.509836 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" event={"ID":"9c1c9053-af28-41e3-b608-69ce5329e3ad","Type":"ContainerDied","Data":"4f78e6de7554abea27ce249c5a69b1c4861ead222cf96d11375679fa0ac9eb5c"} Nov 01 10:17:16.696665 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.696633 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:16.877619 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.877582 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca\") pod \"9c1c9053-af28-41e3-b608-69ce5329e3ad\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " Nov 01 10:17:16.877893 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.877657 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca\") pod \"9c1c9053-af28-41e3-b608-69ce5329e3ad\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " Nov 01 10:17:16.877893 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.877790 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw\" (UniqueName: \"kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw\") pod \"9c1c9053-af28-41e3-b608-69ce5329e3ad\" (UID: \"9c1c9053-af28-41e3-b608-69ce5329e3ad\") " Nov 01 10:17:16.878124 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.878098 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca" pod "9c1c9053-af28-41e3-b608-69ce5329e3ad" (UID: "9c1c9053-af28-41e3-b608-69ce5329e3ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:17:16.878280 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.878258 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:16.880303 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.880263 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca" pod "9c1c9053-af28-41e3-b608-69ce5329e3ad" (UID: "9c1c9053-af28-41e3-b608-69ce5329e3ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:17:16.880916 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.880880 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw" pod "9c1c9053-af28-41e3-b608-69ce5329e3ad" (UID: "9c1c9053-af28-41e3-b608-69ce5329e3ad"). InnerVolumeSpecName "kube-api-access-9qvgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:17:16.979130 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.979061 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvgw\" (UniqueName: \"kubernetes.io/projected/9c1c9053-af28-41e3-b608-69ce5329e3ad-kube-api-access-9qvgw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:16.979130 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:16.979114 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1c9053-af28-41e3-b608-69ce5329e3ad-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:17.542743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:17.538062 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" event={"ID":"9c1c9053-af28-41e3-b608-69ce5329e3ad","Type":"ContainerDied","Data":"18c8d7702309846e820d16d104c2bbd05756d3cdcbdb4328213a23ecf66a035f"} Nov 01 10:17:17.542743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:17.538135 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18c8d7702309846e820d16d104c2bbd05756d3cdcbdb4328213a23ecf66a035f" Nov 01 10:17:17.542743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:17.538216 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj" Nov 01 10:17:23.057412 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:23.057376 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc"] Nov 01 10:17:23.104812 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:23.104768 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fd1ea7-xn6wc"] Nov 01 10:17:25.058057 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:25.058020 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033aa63e-7d21-4968-8d17-fb6ebc95c770" path="/var/lib/kubelet/pods/033aa63e-7d21-4968-8d17-fb6ebc95c770/volumes" Nov 01 10:17:26.078970 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:26.078933 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:17:34.903746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:34.899857 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:17:34.903746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:34.899975 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:17:36.664071 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:36.664034 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:17:45.477032 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:17:45.476994 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:17:48.213544 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.213495 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx"] Nov 01 10:17:48.214130 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.213949 5710 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c9053-af28-41e3-b608-69ce5329e3ad" containerName="revoke-keys" Nov 01 10:17:48.214130 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.213961 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c9053-af28-41e3-b608-69ce5329e3ad" containerName="revoke-keys" Nov 01 10:17:48.214130 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.214059 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c9053-af28-41e3-b608-69ce5329e3ad" containerName="revoke-keys" Nov 01 10:17:48.217155 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.217135 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.271560 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.271523 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx"] Nov 01 10:17:48.274398 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.274365 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29hx\" (UniqueName: \"kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.274589 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.274542 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.274589 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.274570 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.321374 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.321328 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:17:48.321374 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.321348 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:17:48.321374 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.321330 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:17:48.323654 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.323632 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:17:48.376173 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.376135 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.376173 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.376178 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.376416 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.376202 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29hx\" (UniqueName: \"kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.376833 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.376811 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.378916 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.378885 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.445517 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.441362 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29hx\" (UniqueName: \"kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx\") pod \"maas-api-revoke-sub-ae594979-tp5mx\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.531895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.531803 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:48.918558 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:17:48.918524 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4b1a75_734f_43b2_bc9a_b70a963ec21e.slice/crio-b7ed05b2e3f948ab09243e6bdd82a35788abe8ec0f4c8ead1f207510e126e122 WatchSource:0}: Error finding container b7ed05b2e3f948ab09243e6bdd82a35788abe8ec0f4c8ead1f207510e126e122: Status 404 returned error can't find the container with id b7ed05b2e3f948ab09243e6bdd82a35788abe8ec0f4c8ead1f207510e126e122 Nov 01 10:17:48.920513 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:48.920485 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx"] Nov 01 10:17:49.742289 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:49.742243 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" event={"ID":"0b4b1a75-734f-43b2-bc9a-b70a963ec21e","Type":"ContainerStarted","Data":"46c2cd74eecbaa436a70bba10bb7c4d8d93877a9c632789c6964b1de03dbfd98"} Nov 01 10:17:49.742777 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:49.742305 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" event={"ID":"0b4b1a75-734f-43b2-bc9a-b70a963ec21e","Type":"ContainerStarted","Data":"b7ed05b2e3f948ab09243e6bdd82a35788abe8ec0f4c8ead1f207510e126e122"} Nov 01 10:17:49.864171 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:49.864114 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" podStartSLOduration=1.8640957870000001 podStartE2EDuration="1.864095787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:17:49.862309949 +0000 UTC m=+6292.950646621" watchObservedRunningTime="2026-11-01 10:17:49.864095787 +0000 UTC m=+6292.952432458" Nov 01 10:17:50.749053 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:50.749014 5710 generic.go:356] "Generic (PLEG): container finished" podID="0b4b1a75-734f-43b2-bc9a-b70a963ec21e" containerID="46c2cd74eecbaa436a70bba10bb7c4d8d93877a9c632789c6964b1de03dbfd98" exitCode=0 Nov 01 10:17:50.749478 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:50.749085 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" event={"ID":"0b4b1a75-734f-43b2-bc9a-b70a963ec21e","Type":"ContainerDied","Data":"46c2cd74eecbaa436a70bba10bb7c4d8d93877a9c632789c6964b1de03dbfd98"} Nov 01 10:17:51.899256 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:51.899218 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:52.012835 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.012748 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca\") pod \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " Nov 01 10:17:52.012835 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.012802 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx\" (UniqueName: \"kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx\") pod \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " Nov 01 10:17:52.013056 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.012881 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca\") pod \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\" (UID: \"0b4b1a75-734f-43b2-bc9a-b70a963ec21e\") " Nov 01 10:17:52.013125 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.013102 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca" pod "0b4b1a75-734f-43b2-bc9a-b70a963ec21e" (UID: "0b4b1a75-734f-43b2-bc9a-b70a963ec21e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:17:52.013234 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.013207 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:52.015409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.015376 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx" pod "0b4b1a75-734f-43b2-bc9a-b70a963ec21e" (UID: "0b4b1a75-734f-43b2-bc9a-b70a963ec21e"). InnerVolumeSpecName "kube-api-access-n29hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:17:52.015529 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.015447 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca" pod "0b4b1a75-734f-43b2-bc9a-b70a963ec21e" (UID: "0b4b1a75-734f-43b2-bc9a-b70a963ec21e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:17:52.114097 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.114044 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29hx\" (UniqueName: \"kubernetes.io/projected/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-kube-api-access-n29hx\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:52.114097 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.114093 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4b1a75-734f-43b2-bc9a-b70a963ec21e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:17:52.760456 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.760419 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" Nov 01 10:17:52.760690 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.760451 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx" event={"ID":"0b4b1a75-734f-43b2-bc9a-b70a963ec21e","Type":"ContainerDied","Data":"b7ed05b2e3f948ab09243e6bdd82a35788abe8ec0f4c8ead1f207510e126e122"} Nov 01 10:17:52.760690 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:17:52.760501 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ed05b2e3f948ab09243e6bdd82a35788abe8ec0f4c8ead1f207510e126e122" Nov 01 10:18:00.861458 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:00.861419 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:01.508198 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:01.508165 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:18:01.522156 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:01.522131 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:18:06.901134 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:06.901092 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5"] Nov 01 10:18:06.901854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:06.901830 5710 cpu_manager.go:414] "containerMap: removing container" podUID="0b4b1a75-734f-43b2-bc9a-b70a963ec21e" containerName="revoke-keys" Nov 01 10:18:06.901940 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:06.901860 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4b1a75-734f-43b2-bc9a-b70a963ec21e" containerName="revoke-keys" Nov 01 10:18:06.902100 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:06.902019 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4b1a75-734f-43b2-bc9a-b70a963ec21e" containerName="revoke-keys" Nov 01 10:18:06.906303 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:06.906272 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:06.958478 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:06.958430 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5"] Nov 01 10:18:07.008589 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.008551 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:18:07.009796 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.009013 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:18:07.010009 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.009621 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:18:07.012692 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.012668 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:18:07.080535 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.080474 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.080828 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.080547 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfznf\" (UniqueName: \"kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.080828 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.080761 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.188803 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.186377 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.188803 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.186523 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.188803 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.186575 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfznf\" (UniqueName: \"kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.188803 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.188326 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.229748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.215295 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.252754 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.252537 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfznf\" (UniqueName: \"kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf\") pod \"maas-api-revoke-sub-a4aef7d7-h25z5\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:07.523805 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:07.521899 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:08.047524 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:08.047481 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5"] Nov 01 10:18:08.873508 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:08.873462 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" event={"ID":"da7d4e8b-3781-4e59-a4b6-7a1dabaec911","Type":"ContainerStarted","Data":"0c4ac8cf94339a5de4e1cc488904ea978322e68f6924715dd056a588cf09c73d"} Nov 01 10:18:08.873747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:08.873528 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" event={"ID":"da7d4e8b-3781-4e59-a4b6-7a1dabaec911","Type":"ContainerStarted","Data":"20c828fb393044f7045051053e82cb3e90725455367c4c70e413adba1ad402f3"} Nov 01 10:18:09.007744 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:09.006840 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" podStartSLOduration=3.006818383 podStartE2EDuration="3.006818383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:18:09.005163622 +0000 UTC m=+6312.093500298" watchObservedRunningTime="2026-11-01 10:18:09.006818383 +0000 UTC m=+6312.095155056" Nov 01 10:18:09.354700 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:09.353856 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:18:09.354700 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:09.353943 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:18:09.880863 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:09.880823 5710 generic.go:356] "Generic (PLEG): container finished" podID="da7d4e8b-3781-4e59-a4b6-7a1dabaec911" containerID="0c4ac8cf94339a5de4e1cc488904ea978322e68f6924715dd056a588cf09c73d" exitCode=0 Nov 01 10:18:09.881099 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:09.880902 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" event={"ID":"da7d4e8b-3781-4e59-a4b6-7a1dabaec911","Type":"ContainerDied","Data":"0c4ac8cf94339a5de4e1cc488904ea978322e68f6924715dd056a588cf09c73d"} Nov 01 10:18:10.074796 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:10.074763 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:11.029849 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.029815 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:11.176656 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.176557 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca\") pod \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " Nov 01 10:18:11.176656 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.176627 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca\") pod \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " Nov 01 10:18:11.176900 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.176693 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf\" (UniqueName: \"kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf\") pod \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\" (UID: \"da7d4e8b-3781-4e59-a4b6-7a1dabaec911\") " Nov 01 10:18:11.177150 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.177114 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca" pod "da7d4e8b-3781-4e59-a4b6-7a1dabaec911" (UID: "da7d4e8b-3781-4e59-a4b6-7a1dabaec911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:18:11.178975 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.178949 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca" pod "da7d4e8b-3781-4e59-a4b6-7a1dabaec911" (UID: "da7d4e8b-3781-4e59-a4b6-7a1dabaec911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:18:11.178975 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.178958 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf" pod "da7d4e8b-3781-4e59-a4b6-7a1dabaec911" (UID: "da7d4e8b-3781-4e59-a4b6-7a1dabaec911"). InnerVolumeSpecName "kube-api-access-jfznf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:18:11.277522 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.277474 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:11.277522 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.277512 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:11.277522 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.277522 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfznf\" (UniqueName: \"kubernetes.io/projected/da7d4e8b-3781-4e59-a4b6-7a1dabaec911-kube-api-access-jfznf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:11.892035 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.892006 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" Nov 01 10:18:11.892035 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.892032 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5" event={"ID":"da7d4e8b-3781-4e59-a4b6-7a1dabaec911","Type":"ContainerDied","Data":"20c828fb393044f7045051053e82cb3e90725455367c4c70e413adba1ad402f3"} Nov 01 10:18:11.892273 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:11.892077 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c828fb393044f7045051053e82cb3e90725455367c4c70e413adba1ad402f3" Nov 01 10:18:15.477798 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:18:15.477758 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:18:18.182836 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:18.182804 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:26.860168 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:26.860132 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:27.988317 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:27.988269 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb"] Nov 01 10:18:27.988903 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:27.988769 5710 cpu_manager.go:414] "containerMap: removing container" podUID="da7d4e8b-3781-4e59-a4b6-7a1dabaec911" containerName="revoke-keys" Nov 01 10:18:27.988903 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:27.988786 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7d4e8b-3781-4e59-a4b6-7a1dabaec911" containerName="revoke-keys" Nov 01 10:18:27.988903 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:27.988884 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7d4e8b-3781-4e59-a4b6-7a1dabaec911" containerName="revoke-keys" Nov 01 10:18:27.992330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:27.992300 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.048552 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.047502 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb"] Nov 01 10:18:28.059431 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.059391 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbln\" (UniqueName: \"kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.059679 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.059502 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.059679 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.059593 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.095966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.095879 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:18:28.095966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.095909 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:18:28.095966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.095877 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:18:28.095966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.095969 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:18:28.161042 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.160983 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbln\" (UniqueName: \"kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.161286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.161120 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.161286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.161225 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.162409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.162379 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.164107 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.164069 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.227266 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.227211 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbln\" (UniqueName: \"kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln\") pod \"maas-api-revoke-sub-79805ca9-rz8cb\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.307688 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.307586 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:28.768796 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:28.767900 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb"] Nov 01 10:18:29.037836 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:29.030928 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" event={"ID":"6492bfa0-a381-4dea-a722-317238b0f5d2","Type":"ContainerStarted","Data":"c61f3d09561e98dcc5439a03634f4777c69051cc069ec515e2991daf2e107d79"} Nov 01 10:18:30.057189 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:30.057135 5710 generic.go:356] "Generic (PLEG): container finished" podID="6492bfa0-a381-4dea-a722-317238b0f5d2" containerID="2966ed217ea9781dfdbe3d96408d8a302cb134be875134f2205af1876cd945b4" exitCode=0 Nov 01 10:18:30.057870 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:30.057297 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" event={"ID":"6492bfa0-a381-4dea-a722-317238b0f5d2","Type":"ContainerDied","Data":"2966ed217ea9781dfdbe3d96408d8a302cb134be875134f2205af1876cd945b4"} Nov 01 10:18:31.271579 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.271548 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:31.355771 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.355704 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln\" (UniqueName: \"kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln\") pod \"6492bfa0-a381-4dea-a722-317238b0f5d2\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " Nov 01 10:18:31.355977 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.355846 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca\") pod \"6492bfa0-a381-4dea-a722-317238b0f5d2\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " Nov 01 10:18:31.355977 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.355948 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca\") pod \"6492bfa0-a381-4dea-a722-317238b0f5d2\" (UID: \"6492bfa0-a381-4dea-a722-317238b0f5d2\") " Nov 01 10:18:31.356318 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.356292 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca" pod "6492bfa0-a381-4dea-a722-317238b0f5d2" (UID: "6492bfa0-a381-4dea-a722-317238b0f5d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:18:31.358275 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.358252 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca" pod "6492bfa0-a381-4dea-a722-317238b0f5d2" (UID: "6492bfa0-a381-4dea-a722-317238b0f5d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:18:31.358342 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.358293 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln" pod "6492bfa0-a381-4dea-a722-317238b0f5d2" (UID: "6492bfa0-a381-4dea-a722-317238b0f5d2"). InnerVolumeSpecName "kube-api-access-zbbln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:18:31.457252 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.457210 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:31.457252 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.457244 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbln\" (UniqueName: \"kubernetes.io/projected/6492bfa0-a381-4dea-a722-317238b0f5d2-kube-api-access-zbbln\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:31.457252 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:31.457254 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6492bfa0-a381-4dea-a722-317238b0f5d2-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:32.072538 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:32.072501 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" event={"ID":"6492bfa0-a381-4dea-a722-317238b0f5d2","Type":"ContainerDied","Data":"c61f3d09561e98dcc5439a03634f4777c69051cc069ec515e2991daf2e107d79"} Nov 01 10:18:32.072792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:32.072555 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c61f3d09561e98dcc5439a03634f4777c69051cc069ec515e2991daf2e107d79" Nov 01 10:18:32.072792 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:32.072516 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb" Nov 01 10:18:35.890838 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:35.890798 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:44.592876 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:44.592836 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:45.394239 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.394197 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz"] Nov 01 10:18:45.394922 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.394896 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6492bfa0-a381-4dea-a722-317238b0f5d2" containerName="revoke-keys" Nov 01 10:18:45.394922 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.394922 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6492bfa0-a381-4dea-a722-317238b0f5d2" containerName="revoke-keys" Nov 01 10:18:45.395239 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.395084 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6492bfa0-a381-4dea-a722-317238b0f5d2" containerName="revoke-keys" Nov 01 10:18:45.399016 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.398985 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.409295 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.408973 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.409295 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.409072 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w9v\" (UniqueName: \"kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.409295 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.409143 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.453027 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.452982 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz"] Nov 01 10:18:45.492554 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:18:45.492510 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:18:45.501711 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.501656 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:18:45.501711 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.501675 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:18:45.501711 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.501666 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:18:45.502025 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.501707 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:18:45.510033 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.509990 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.510261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.510093 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.510261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.510180 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w9v\" (UniqueName: \"kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.511105 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.511055 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.512903 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.512876 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.574523 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.574477 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w9v\" (UniqueName: \"kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v\") pod \"maas-api-revoke-sub-0b7e8edb-8bdtz\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:45.718781 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:45.718657 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:46.121392 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:18:46.121354 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685651e8_112b_4121_a95a_e8ec5d5e9557.slice/crio-0abcc8ec85af4351c806061e33408b445f24b424adb63ec642232b3fa48a85a1 WatchSource:0}: Error finding container 0abcc8ec85af4351c806061e33408b445f24b424adb63ec642232b3fa48a85a1: Status 404 returned error can't find the container with id 0abcc8ec85af4351c806061e33408b445f24b424adb63ec642232b3fa48a85a1 Nov 01 10:18:46.122409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:46.122329 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz"] Nov 01 10:18:46.161031 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:46.160981 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" event={"ID":"685651e8-112b-4121-a95a-e8ec5d5e9557","Type":"ContainerStarted","Data":"0abcc8ec85af4351c806061e33408b445f24b424adb63ec642232b3fa48a85a1"} Nov 01 10:18:47.167646 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:47.167610 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" event={"ID":"685651e8-112b-4121-a95a-e8ec5d5e9557","Type":"ContainerStarted","Data":"866abf4e5c8b060a8ad35d1382c6a401f2826f9ed5dfddba351ffeea7fc31d56"} Nov 01 10:18:48.174898 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:48.174854 5710 generic.go:356] "Generic (PLEG): container finished" podID="685651e8-112b-4121-a95a-e8ec5d5e9557" containerID="866abf4e5c8b060a8ad35d1382c6a401f2826f9ed5dfddba351ffeea7fc31d56" exitCode=0 Nov 01 10:18:48.175395 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:48.174942 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" event={"ID":"685651e8-112b-4121-a95a-e8ec5d5e9557","Type":"ContainerDied","Data":"866abf4e5c8b060a8ad35d1382c6a401f2826f9ed5dfddba351ffeea7fc31d56"} Nov 01 10:18:49.325605 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.325572 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:49.382507 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.382468 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca\") pod \"685651e8-112b-4121-a95a-e8ec5d5e9557\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " Nov 01 10:18:49.382753 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.382592 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v\" (UniqueName: \"kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v\") pod \"685651e8-112b-4121-a95a-e8ec5d5e9557\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " Nov 01 10:18:49.382753 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.382681 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca\") pod \"685651e8-112b-4121-a95a-e8ec5d5e9557\" (UID: \"685651e8-112b-4121-a95a-e8ec5d5e9557\") " Nov 01 10:18:49.382912 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.382882 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca" pod "685651e8-112b-4121-a95a-e8ec5d5e9557" (UID: "685651e8-112b-4121-a95a-e8ec5d5e9557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:18:49.383035 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.383017 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:49.384979 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.384954 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca" pod "685651e8-112b-4121-a95a-e8ec5d5e9557" (UID: "685651e8-112b-4121-a95a-e8ec5d5e9557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:18:49.384979 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.384963 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v" pod "685651e8-112b-4121-a95a-e8ec5d5e9557" (UID: "685651e8-112b-4121-a95a-e8ec5d5e9557"). InnerVolumeSpecName "kube-api-access-r5w9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:18:49.484353 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.484251 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685651e8-112b-4121-a95a-e8ec5d5e9557-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:49.484353 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:49.484289 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w9v\" (UniqueName: \"kubernetes.io/projected/685651e8-112b-4121-a95a-e8ec5d5e9557-kube-api-access-r5w9v\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:50.186225 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:50.186198 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" Nov 01 10:18:50.186435 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:50.186199 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz" event={"ID":"685651e8-112b-4121-a95a-e8ec5d5e9557","Type":"ContainerDied","Data":"0abcc8ec85af4351c806061e33408b445f24b424adb63ec642232b3fa48a85a1"} Nov 01 10:18:50.186435 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:50.186332 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0abcc8ec85af4351c806061e33408b445f24b424adb63ec642232b3fa48a85a1" Nov 01 10:18:53.595535 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:53.595493 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:18:56.637341 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.637297 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447"] Nov 01 10:18:56.638052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.638029 5710 cpu_manager.go:414] "containerMap: removing container" podUID="685651e8-112b-4121-a95a-e8ec5d5e9557" containerName="revoke-keys" Nov 01 10:18:56.638110 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.638056 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685651e8-112b-4121-a95a-e8ec5d5e9557" containerName="revoke-keys" Nov 01 10:18:56.638230 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.638217 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="685651e8-112b-4121-a95a-e8ec5d5e9557" containerName="revoke-keys" Nov 01 10:18:56.642462 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.642437 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.695008 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.694970 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447"] Nov 01 10:18:56.744296 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.744250 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:18:56.744296 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.744276 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:18:56.744553 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.744339 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:18:56.748759 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.748707 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:18:56.758259 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.758232 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whz4v\" (UniqueName: \"kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.758364 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.758266 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.758364 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.758347 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.859561 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.859524 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whz4v\" (UniqueName: \"kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.859561 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.859566 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.859844 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.859607 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.860269 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.860249 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.862243 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.862222 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.923852 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.923763 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whz4v\" (UniqueName: \"kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v\") pod \"maas-api-revoke-sub-3d4156f1-ps447\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:56.958854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:56.958808 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:57.337819 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:18:57.337785 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd86c12_86b2_47fc_9849_8f0aafeac4c0.slice/crio-2a36eae27b5e86ffbe03e33fad6a1c7fbd2149821f0b2a8e937f3eca5cab87a8 WatchSource:0}: Error finding container 2a36eae27b5e86ffbe03e33fad6a1c7fbd2149821f0b2a8e937f3eca5cab87a8: Status 404 returned error can't find the container with id 2a36eae27b5e86ffbe03e33fad6a1c7fbd2149821f0b2a8e937f3eca5cab87a8 Nov 01 10:18:57.339556 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:57.339529 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447"] Nov 01 10:18:58.233709 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:58.233660 5710 generic.go:356] "Generic (PLEG): container finished" podID="4cd86c12-86b2-47fc-9849-8f0aafeac4c0" containerID="287f4e264a0e06a0123ea6e7c92cc9c277f74107105a0a250f0ad4d6b1daad1e" exitCode=0 Nov 01 10:18:58.234252 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:58.233798 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" event={"ID":"4cd86c12-86b2-47fc-9849-8f0aafeac4c0","Type":"ContainerDied","Data":"287f4e264a0e06a0123ea6e7c92cc9c277f74107105a0a250f0ad4d6b1daad1e"} Nov 01 10:18:58.234252 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:58.233901 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" event={"ID":"4cd86c12-86b2-47fc-9849-8f0aafeac4c0","Type":"ContainerStarted","Data":"2a36eae27b5e86ffbe03e33fad6a1c7fbd2149821f0b2a8e937f3eca5cab87a8"} Nov 01 10:18:59.384370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.384329 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:18:59.485936 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.485896 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v\" (UniqueName: \"kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v\") pod \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " Nov 01 10:18:59.486152 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.486055 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca\") pod \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " Nov 01 10:18:59.486152 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.486092 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca\") pod \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\" (UID: \"4cd86c12-86b2-47fc-9849-8f0aafeac4c0\") " Nov 01 10:18:59.486536 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.486509 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca" pod "4cd86c12-86b2-47fc-9849-8f0aafeac4c0" (UID: "4cd86c12-86b2-47fc-9849-8f0aafeac4c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:18:59.488324 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.488303 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca" pod "4cd86c12-86b2-47fc-9849-8f0aafeac4c0" (UID: "4cd86c12-86b2-47fc-9849-8f0aafeac4c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:18:59.488324 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.488313 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v" pod "4cd86c12-86b2-47fc-9849-8f0aafeac4c0" (UID: "4cd86c12-86b2-47fc-9849-8f0aafeac4c0"). InnerVolumeSpecName "kube-api-access-whz4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:18:59.587703 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.587669 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:59.587703 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.587700 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:18:59.587703 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:18:59.587710 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whz4v\" (UniqueName: \"kubernetes.io/projected/4cd86c12-86b2-47fc-9849-8f0aafeac4c0-kube-api-access-whz4v\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:19:00.246409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:00.246374 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" event={"ID":"4cd86c12-86b2-47fc-9849-8f0aafeac4c0","Type":"ContainerDied","Data":"2a36eae27b5e86ffbe03e33fad6a1c7fbd2149821f0b2a8e937f3eca5cab87a8"} Nov 01 10:19:00.246604 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:00.246426 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a36eae27b5e86ffbe03e33fad6a1c7fbd2149821f0b2a8e937f3eca5cab87a8" Nov 01 10:19:00.246604 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:00.246390 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447" Nov 01 10:19:01.309114 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:01.308996 5710 scope.go:122] "RemoveContainer" containerID="21065f2dedf0597ef203575946595b6862c520f2aa0e8afe705fe78bc053dc89" Nov 01 10:19:11.067661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:11.067625 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:19:15.478405 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:19:15.478364 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:19:19.901748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:19.897863 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:19:19.901748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:19.897965 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:19:21.309744 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.308992 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447"] Nov 01 10:19:21.360036 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.359998 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ps447"] Nov 01 10:19:21.428840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.428801 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww"] Nov 01 10:19:21.429242 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.429231 5710 cpu_manager.go:414] "containerMap: removing container" podUID="4cd86c12-86b2-47fc-9849-8f0aafeac4c0" containerName="revoke-keys" Nov 01 10:19:21.429343 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.429244 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd86c12-86b2-47fc-9849-8f0aafeac4c0" containerName="revoke-keys" Nov 01 10:19:21.429343 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.429320 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd86c12-86b2-47fc-9849-8f0aafeac4c0" containerName="revoke-keys" Nov 01 10:19:21.432469 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.432450 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.476387 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.476346 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb79x\" (UniqueName: \"kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.476619 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.476475 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.476619 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.476500 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.487218 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.487181 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww"] Nov 01 10:19:21.489473 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.489437 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:19:21.489631 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.489500 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:19:21.489631 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.489580 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:19:21.519975 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.519941 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:19:21.577898 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.577799 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.577898 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.577862 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.578173 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.577981 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb79x\" (UniqueName: \"kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.578624 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.578601 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.580641 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.580610 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.584438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.584416 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:19:21.644608 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.644578 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb79x\" (UniqueName: \"kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x\") pod \"maas-api-revoke-sub-3d4156f1-pl2ww\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:21.745601 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:21.745557 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:22.130085 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:19:22.130040 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229aa8f0_54fd_4c86_8eba_c8fbe037f7c4.slice/crio-b203f80074c939ffbe6e2efcdf75e61906b3bf03d41ad272693bec2ce1b21915 WatchSource:0}: Error finding container b203f80074c939ffbe6e2efcdf75e61906b3bf03d41ad272693bec2ce1b21915: Status 404 returned error can't find the container with id b203f80074c939ffbe6e2efcdf75e61906b3bf03d41ad272693bec2ce1b21915 Nov 01 10:19:22.131246 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:22.131201 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww"] Nov 01 10:19:22.471195 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:22.471151 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" event={"ID":"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4","Type":"ContainerStarted","Data":"35c9ff9a73c4a7e85f6326af97a0821acedbe18e8ac642b8f929661988058511"} Nov 01 10:19:22.471641 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:22.471257 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" event={"ID":"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4","Type":"ContainerStarted","Data":"b203f80074c939ffbe6e2efcdf75e61906b3bf03d41ad272693bec2ce1b21915"} Nov 01 10:19:22.594412 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:22.594357 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" podStartSLOduration=1.5943396669999998 podStartE2EDuration="1.594339667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:19:22.592216138 +0000 UTC m=+6385.680552809" watchObservedRunningTime="2026-11-01 10:19:22.594339667 +0000 UTC m=+6385.682676340" Nov 01 10:19:23.056906 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:23.056868 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd86c12-86b2-47fc-9849-8f0aafeac4c0" path="/var/lib/kubelet/pods/4cd86c12-86b2-47fc-9849-8f0aafeac4c0/volumes" Nov 01 10:19:23.478317 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:23.478284 5710 generic.go:356] "Generic (PLEG): container finished" podID="229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" containerID="35c9ff9a73c4a7e85f6326af97a0821acedbe18e8ac642b8f929661988058511" exitCode=0 Nov 01 10:19:23.478848 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:23.478370 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" event={"ID":"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4","Type":"ContainerDied","Data":"35c9ff9a73c4a7e85f6326af97a0821acedbe18e8ac642b8f929661988058511"} Nov 01 10:19:24.627555 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.627526 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:24.710261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.710225 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca\") pod \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " Nov 01 10:19:24.710481 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.710348 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x\" (UniqueName: \"kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x\") pod \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " Nov 01 10:19:24.710481 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.710376 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca\") pod \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\" (UID: \"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4\") " Nov 01 10:19:24.710790 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.710768 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca" pod "229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" (UID: "229aa8f0-54fd-4c86-8eba-c8fbe037f7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:19:24.712586 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.712561 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca" pod "229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" (UID: "229aa8f0-54fd-4c86-8eba-c8fbe037f7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:19:24.712930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.712910 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x" pod "229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" (UID: "229aa8f0-54fd-4c86-8eba-c8fbe037f7c4"). InnerVolumeSpecName "kube-api-access-jb79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:19:24.811915 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.811806 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb79x\" (UniqueName: \"kubernetes.io/projected/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-kube-api-access-jb79x\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:19:24.811915 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.811850 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:19:24.811915 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:24.811866 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:19:25.489618 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:25.489584 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" Nov 01 10:19:25.489618 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:25.489601 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww" event={"ID":"229aa8f0-54fd-4c86-8eba-c8fbe037f7c4","Type":"ContainerDied","Data":"b203f80074c939ffbe6e2efcdf75e61906b3bf03d41ad272693bec2ce1b21915"} Nov 01 10:19:25.489886 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:25.489655 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b203f80074c939ffbe6e2efcdf75e61906b3bf03d41ad272693bec2ce1b21915" Nov 01 10:19:35.780928 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:35.780841 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:19:45.372369 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:45.372329 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:19:45.476825 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:19:45.476781 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:19:53.461927 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:53.461888 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:19:57.656955 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:57.656898 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:19:57.657531 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:19:57.657011 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:20:00.614611 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.614549 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb"] Nov 01 10:20:00.615370 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.615347 5710 cpu_manager.go:414] "containerMap: removing container" podUID="229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" containerName="revoke-keys" Nov 01 10:20:00.615670 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.615374 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" containerName="revoke-keys" Nov 01 10:20:00.615670 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.615502 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" containerName="revoke-keys" Nov 01 10:20:00.619747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.619707 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.672474 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.672439 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb"] Nov 01 10:20:00.707929 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.707896 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.708140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.707948 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.708140 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.708073 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjscc\" (UniqueName: \"kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.721496 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.721445 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:20:00.721496 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.721484 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:20:00.721839 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.721671 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:20:00.725904 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.725876 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:20:00.809161 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.809112 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.809367 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.809173 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.809367 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.809200 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjscc\" (UniqueName: \"kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.809829 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.809808 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.811816 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.811783 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.873198 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.873117 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjscc\" (UniqueName: \"kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc\") pod \"maas-api-revoke-sub-e2fc9c5b-mx7sb\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:00.933886 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:00.933850 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:01.322672 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:20:01.322640 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ca5489_6277_4974_8387_0194782e21ad.slice/crio-1be606b77e02bfbcde28f5641d9a4f468e27d3d3405df4cf89d3e6ead0d9de2b WatchSource:0}: Error finding container 1be606b77e02bfbcde28f5641d9a4f468e27d3d3405df4cf89d3e6ead0d9de2b: Status 404 returned error can't find the container with id 1be606b77e02bfbcde28f5641d9a4f468e27d3d3405df4cf89d3e6ead0d9de2b Nov 01 10:20:01.326064 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:01.326035 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb"] Nov 01 10:20:01.698000 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:01.697962 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" event={"ID":"b3ca5489-6277-4974-8387-0194782e21ad","Type":"ContainerStarted","Data":"99fb4d58df9d508f5fda3fc42cb91844f5d996c58c3d9f2d40841cce3a4786de"} Nov 01 10:20:01.698468 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:01.698021 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" event={"ID":"b3ca5489-6277-4974-8387-0194782e21ad","Type":"ContainerStarted","Data":"1be606b77e02bfbcde28f5641d9a4f468e27d3d3405df4cf89d3e6ead0d9de2b"} Nov 01 10:20:01.820759 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:01.820675 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" podStartSLOduration=1.820653321 podStartE2EDuration="1.820653321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:20:01.818087601 +0000 UTC m=+6424.906424275" watchObservedRunningTime="2026-11-01 10:20:01.820653321 +0000 UTC m=+6424.908989994" Nov 01 10:20:02.704661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:02.704625 5710 generic.go:356] "Generic (PLEG): container finished" podID="b3ca5489-6277-4974-8387-0194782e21ad" containerID="99fb4d58df9d508f5fda3fc42cb91844f5d996c58c3d9f2d40841cce3a4786de" exitCode=0 Nov 01 10:20:02.704661 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:02.704665 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" event={"ID":"b3ca5489-6277-4974-8387-0194782e21ad","Type":"ContainerDied","Data":"99fb4d58df9d508f5fda3fc42cb91844f5d996c58c3d9f2d40841cce3a4786de"} Nov 01 10:20:03.863438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.863411 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:03.943356 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.943322 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca\") pod \"b3ca5489-6277-4974-8387-0194782e21ad\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " Nov 01 10:20:03.943583 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.943362 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc\" (UniqueName: \"kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc\") pod \"b3ca5489-6277-4974-8387-0194782e21ad\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " Nov 01 10:20:03.943583 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.943417 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca\") pod \"b3ca5489-6277-4974-8387-0194782e21ad\" (UID: \"b3ca5489-6277-4974-8387-0194782e21ad\") " Nov 01 10:20:03.943826 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.943756 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca" pod "b3ca5489-6277-4974-8387-0194782e21ad" (UID: "b3ca5489-6277-4974-8387-0194782e21ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:20:03.943899 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.943876 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:03.945707 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.945680 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca" pod "b3ca5489-6277-4974-8387-0194782e21ad" (UID: "b3ca5489-6277-4974-8387-0194782e21ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:20:03.945876 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:03.945816 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc" pod "b3ca5489-6277-4974-8387-0194782e21ad" (UID: "b3ca5489-6277-4974-8387-0194782e21ad"). InnerVolumeSpecName "kube-api-access-cjscc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:20:04.045021 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:04.044923 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjscc\" (UniqueName: \"kubernetes.io/projected/b3ca5489-6277-4974-8387-0194782e21ad-kube-api-access-cjscc\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:04.045021 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:04.044959 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ca5489-6277-4974-8387-0194782e21ad-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:04.718466 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:04.718438 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" Nov 01 10:20:04.718663 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:04.718471 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb" event={"ID":"b3ca5489-6277-4974-8387-0194782e21ad","Type":"ContainerDied","Data":"1be606b77e02bfbcde28f5641d9a4f468e27d3d3405df4cf89d3e6ead0d9de2b"} Nov 01 10:20:04.718663 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:04.718519 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1be606b77e02bfbcde28f5641d9a4f468e27d3d3405df4cf89d3e6ead0d9de2b" Nov 01 10:20:11.342112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.342079 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk"] Nov 01 10:20:11.342645 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.342493 5710 cpu_manager.go:414] "containerMap: removing container" podUID="b3ca5489-6277-4974-8387-0194782e21ad" containerName="revoke-keys" Nov 01 10:20:11.342645 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.342504 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ca5489-6277-4974-8387-0194782e21ad" containerName="revoke-keys" Nov 01 10:20:11.342645 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.342595 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ca5489-6277-4974-8387-0194782e21ad" containerName="revoke-keys" Nov 01 10:20:11.346091 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.346072 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.400939 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.400900 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk"] Nov 01 10:20:11.423077 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.423041 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668wl\" (UniqueName: \"kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.423278 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.423105 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.423278 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.423191 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.450438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.450402 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:20:11.450438 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.450404 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:20:11.453196 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.453176 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:20:11.453318 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.453297 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:20:11.524845 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.524794 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668wl\" (UniqueName: \"kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.525082 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.524862 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.525082 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.524893 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.525506 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.525485 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.527527 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.527507 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.589733 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.589688 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668wl\" (UniqueName: \"kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl\") pod \"maas-api-revoke-sub-5a29efb3-btkkk\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:11.660216 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:11.660129 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:12.035339 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:20:12.035293 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604f74a1_fe23_48cd_b7b5_3151fda3ce28.slice/crio-b59d61f3013bc7af1d7cdb913b4c502ba1aeb83185367725325eccd9eda9a4be WatchSource:0}: Error finding container b59d61f3013bc7af1d7cdb913b4c502ba1aeb83185367725325eccd9eda9a4be: Status 404 returned error can't find the container with id b59d61f3013bc7af1d7cdb913b4c502ba1aeb83185367725325eccd9eda9a4be Nov 01 10:20:12.037075 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:12.037052 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk"] Nov 01 10:20:12.769007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:12.768956 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" event={"ID":"604f74a1-fe23-48cd-b7b5-3151fda3ce28","Type":"ContainerStarted","Data":"0cdc8bd554eb33b620135f5bb66accdabc4e0f1508eaf8f2a70e180e3eb19e1f"} Nov 01 10:20:12.769467 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:12.769015 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" event={"ID":"604f74a1-fe23-48cd-b7b5-3151fda3ce28","Type":"ContainerStarted","Data":"b59d61f3013bc7af1d7cdb913b4c502ba1aeb83185367725325eccd9eda9a4be"} Nov 01 10:20:12.890978 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:12.890922 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" podStartSLOduration=1.890906766 podStartE2EDuration="1.890906766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:20:12.889408907 +0000 UTC m=+6435.977745578" watchObservedRunningTime="2026-11-01 10:20:12.890906766 +0000 UTC m=+6435.979243438" Nov 01 10:20:13.351306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:13.351268 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:20:13.777629 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:13.777516 5710 generic.go:356] "Generic (PLEG): container finished" podID="604f74a1-fe23-48cd-b7b5-3151fda3ce28" containerID="0cdc8bd554eb33b620135f5bb66accdabc4e0f1508eaf8f2a70e180e3eb19e1f" exitCode=0 Nov 01 10:20:13.778210 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:13.777614 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" event={"ID":"604f74a1-fe23-48cd-b7b5-3151fda3ce28","Type":"ContainerDied","Data":"0cdc8bd554eb33b620135f5bb66accdabc4e0f1508eaf8f2a70e180e3eb19e1f"} Nov 01 10:20:15.119911 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.111933 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:15.210221 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.210177 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca\") pod \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " Nov 01 10:20:15.211078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.211049 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca\") pod \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " Nov 01 10:20:15.211338 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.211314 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl\" (UniqueName: \"kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl\") pod \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\" (UID: \"604f74a1-fe23-48cd-b7b5-3151fda3ce28\") " Nov 01 10:20:15.212332 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.212308 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca" pod "604f74a1-fe23-48cd-b7b5-3151fda3ce28" (UID: "604f74a1-fe23-48cd-b7b5-3151fda3ce28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:20:15.227190 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.227149 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca" pod "604f74a1-fe23-48cd-b7b5-3151fda3ce28" (UID: "604f74a1-fe23-48cd-b7b5-3151fda3ce28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:20:15.239324 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.239281 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl" pod "604f74a1-fe23-48cd-b7b5-3151fda3ce28" (UID: "604f74a1-fe23-48cd-b7b5-3151fda3ce28"). InnerVolumeSpecName "kube-api-access-668wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:20:15.318446 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.318398 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:15.318765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.318741 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668wl\" (UniqueName: \"kubernetes.io/projected/604f74a1-fe23-48cd-b7b5-3151fda3ce28-kube-api-access-668wl\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:15.343831 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.341784 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604f74a1-fe23-48cd-b7b5-3151fda3ce28-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:15.491745 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:20:15.489013 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:20:15.827515 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.827405 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" event={"ID":"604f74a1-fe23-48cd-b7b5-3151fda3ce28","Type":"ContainerDied","Data":"b59d61f3013bc7af1d7cdb913b4c502ba1aeb83185367725325eccd9eda9a4be"} Nov 01 10:20:15.827845 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.827817 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b59d61f3013bc7af1d7cdb913b4c502ba1aeb83185367725325eccd9eda9a4be" Nov 01 10:20:15.828094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:15.828076 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk" Nov 01 10:20:22.053007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:22.052966 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn"] Nov 01 10:20:22.104442 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:22.104407 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-chlbn"] Nov 01 10:20:23.057256 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:23.057221 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf" path="/var/lib/kubelet/pods/cdfc4516-bdd1-433a-a4ca-7d1ab5779dcf/volumes" Nov 01 10:20:23.995123 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:23.995087 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:20:27.914944 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:27.914910 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww"] Nov 01 10:20:27.966822 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:27.966785 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl2ww"] Nov 01 10:20:28.035108 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.035067 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz"] Nov 01 10:20:28.035563 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.035548 5710 cpu_manager.go:414] "containerMap: removing container" podUID="604f74a1-fe23-48cd-b7b5-3151fda3ce28" containerName="revoke-keys" Nov 01 10:20:28.035615 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.035565 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604f74a1-fe23-48cd-b7b5-3151fda3ce28" containerName="revoke-keys" Nov 01 10:20:28.035667 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.035656 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="604f74a1-fe23-48cd-b7b5-3151fda3ce28" containerName="revoke-keys" Nov 01 10:20:28.039273 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.039254 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.082040 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.082001 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.082274 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.082070 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.082274 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.082181 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxl7\" (UniqueName: \"kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.093980 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.093946 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:20:28.094226 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.094203 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:20:28.094404 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.094371 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:20:28.094473 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.094442 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:20:28.095779 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.095755 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz"] Nov 01 10:20:28.183366 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.183269 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.183576 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.183385 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.183576 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.183428 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxl7\" (UniqueName: \"kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.184118 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.184095 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.186091 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.186066 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.247590 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.247550 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxl7\" (UniqueName: \"kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7\") pod \"maas-api-revoke-sub-3d4156f1-qq5wz\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.352481 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.352442 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:28.730942 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:20:28.730911 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064c8670_1d0f_47db_a91c_d0651cb27789.slice/crio-bb55c9a6e4ccd652be5fe51ed287f8ea21808ac2d07a76fd6eea9bc1358a9bb5 WatchSource:0}: Error finding container bb55c9a6e4ccd652be5fe51ed287f8ea21808ac2d07a76fd6eea9bc1358a9bb5: Status 404 returned error can't find the container with id bb55c9a6e4ccd652be5fe51ed287f8ea21808ac2d07a76fd6eea9bc1358a9bb5 Nov 01 10:20:28.732254 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.732230 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz"] Nov 01 10:20:28.917280 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:28.917236 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" event={"ID":"064c8670-1d0f-47db-a91c-d0651cb27789","Type":"ContainerStarted","Data":"bb55c9a6e4ccd652be5fe51ed287f8ea21808ac2d07a76fd6eea9bc1358a9bb5"} Nov 01 10:20:29.057873 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:29.057839 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229aa8f0-54fd-4c86-8eba-c8fbe037f7c4" path="/var/lib/kubelet/pods/229aa8f0-54fd-4c86-8eba-c8fbe037f7c4/volumes" Nov 01 10:20:29.924332 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:29.924296 5710 generic.go:356] "Generic (PLEG): container finished" podID="064c8670-1d0f-47db-a91c-d0651cb27789" containerID="5e9632b94996120e0d3a96aa18ea5bd82ba6b0fc8fc95a84396a0233d4aaac4e" exitCode=0 Nov 01 10:20:29.924886 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:29.924422 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" event={"ID":"064c8670-1d0f-47db-a91c-d0651cb27789","Type":"ContainerDied","Data":"5e9632b94996120e0d3a96aa18ea5bd82ba6b0fc8fc95a84396a0233d4aaac4e"} Nov 01 10:20:31.089570 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.089540 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:31.215014 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.214916 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7\" (UniqueName: \"kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7\") pod \"064c8670-1d0f-47db-a91c-d0651cb27789\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " Nov 01 10:20:31.215204 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.215015 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca\") pod \"064c8670-1d0f-47db-a91c-d0651cb27789\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " Nov 01 10:20:31.215204 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.215047 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca\") pod \"064c8670-1d0f-47db-a91c-d0651cb27789\" (UID: \"064c8670-1d0f-47db-a91c-d0651cb27789\") " Nov 01 10:20:31.215404 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.215379 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca" pod "064c8670-1d0f-47db-a91c-d0651cb27789" (UID: "064c8670-1d0f-47db-a91c-d0651cb27789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:20:31.215518 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.215502 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:31.217269 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.217238 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca" pod "064c8670-1d0f-47db-a91c-d0651cb27789" (UID: "064c8670-1d0f-47db-a91c-d0651cb27789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:20:31.217397 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.217332 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7" pod "064c8670-1d0f-47db-a91c-d0651cb27789" (UID: "064c8670-1d0f-47db-a91c-d0651cb27789"). InnerVolumeSpecName "kube-api-access-jnxl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:20:31.318169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.318130 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxl7\" (UniqueName: \"kubernetes.io/projected/064c8670-1d0f-47db-a91c-d0651cb27789-kube-api-access-jnxl7\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:31.318169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.318166 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064c8670-1d0f-47db-a91c-d0651cb27789-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:20:31.938578 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.938538 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" Nov 01 10:20:31.938578 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.938570 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz" event={"ID":"064c8670-1d0f-47db-a91c-d0651cb27789","Type":"ContainerDied","Data":"bb55c9a6e4ccd652be5fe51ed287f8ea21808ac2d07a76fd6eea9bc1358a9bb5"} Nov 01 10:20:31.938933 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:31.938621 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb55c9a6e4ccd652be5fe51ed287f8ea21808ac2d07a76fd6eea9bc1358a9bb5" Nov 01 10:20:42.388358 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:42.388320 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:20:44.650741 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.649995 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2"] Nov 01 10:20:44.650741 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.650685 5710 cpu_manager.go:414] "containerMap: removing container" podUID="064c8670-1d0f-47db-a91c-d0651cb27789" containerName="revoke-keys" Nov 01 10:20:44.667002 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.650707 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064c8670-1d0f-47db-a91c-d0651cb27789" containerName="revoke-keys" Nov 01 10:20:44.667002 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.663911 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="064c8670-1d0f-47db-a91c-d0651cb27789" containerName="revoke-keys" Nov 01 10:20:44.680197 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.677062 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.744783 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.733993 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-23b415-openshift-default-dockercfg-n55t4\"" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774280 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774355 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774469 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774497 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774526 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gld5\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774558 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774589 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774631 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.790757 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.774663 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.847050 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.847007 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2"] Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876310 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876368 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876396 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gld5\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876427 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876460 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876507 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876536 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876589 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.876963 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.876636 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.879573 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.878309 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.879573 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.878576 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.879573 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.879159 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.880603 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.880580 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.892744 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.881035 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.902455 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.902375 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.969007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.967478 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.969007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.968247 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gld5\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:44.969007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:44.968920 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token\") pod \"e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:45.037965 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:45.037926 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:45.538914 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:45.525606 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2"] Nov 01 10:20:45.538914 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:20:45.534487 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:20:45.559833 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:45.559784 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:20:45.559967 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:45.559878 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:20:45.559967 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:45.559923 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:20:46.075484 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:46.075437 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" event={"ID":"cc808c31-25df-476c-8c57-3cc1eb4a3b49","Type":"ContainerStarted","Data":"c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00"} Nov 01 10:20:46.075942 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:46.075545 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" event={"ID":"cc808c31-25df-476c-8c57-3cc1eb4a3b49","Type":"ContainerStarted","Data":"33efe6ba8e2d8abab80b7e8d8075ccd17cece7b2c1bd1126dbeb5cc6c908d2ad"} Nov 01 10:20:46.254462 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:46.254401 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podStartSLOduration=2.254386853 podStartE2EDuration="2.254386853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:20:46.253294755 +0000 UTC m=+6469.341631437" watchObservedRunningTime="2026-11-01 10:20:46.254386853 +0000 UTC m=+6469.342723523" Nov 01 10:20:47.038920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:47.038876 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:47.044445 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:47.044418 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:47.084102 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:47.084063 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:47.085078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:47.085060 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:20:52.916999 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:52.916955 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c"] Nov 01 10:20:52.920701 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:52.920671 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:52.972352 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:52.972308 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvphr\" (UniqueName: \"kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr\") pod \"payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:52.972591 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:52.972451 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:52.975516 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:52.975484 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c"] Nov 01 10:20:53.024969 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.024935 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e45b1a16-dockercfg-lgfjk\"" Nov 01 10:20:53.024969 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.024953 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-23b415\"" Nov 01 10:20:53.074095 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.074052 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvphr\" (UniqueName: \"kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr\") pod \"payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:53.074312 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.074126 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:53.074840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.074821 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:53.139037 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.138998 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvphr\" (UniqueName: \"kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr\") pod \"payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:53.235368 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.235275 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:53.609501 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:20:53.609458 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3046a42c_fb23_41f3_9ee2_56466bab3a21.slice/crio-07526fae02df6428b231ebca265fbe6dd1dc7914562625899d3153fbae280089 WatchSource:0}: Error finding container 07526fae02df6428b231ebca265fbe6dd1dc7914562625899d3153fbae280089: Status 404 returned error can't find the container with id 07526fae02df6428b231ebca265fbe6dd1dc7914562625899d3153fbae280089 Nov 01 10:20:53.613450 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:53.613421 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c"] Nov 01 10:20:54.124924 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:54.124883 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" event={"ID":"3046a42c-fb23-41f3-9ee2-56466bab3a21","Type":"ContainerStarted","Data":"188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd"} Nov 01 10:20:54.125364 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:54.124947 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" event={"ID":"3046a42c-fb23-41f3-9ee2-56466bab3a21","Type":"ContainerStarted","Data":"07526fae02df6428b231ebca265fbe6dd1dc7914562625899d3153fbae280089"} Nov 01 10:20:54.125364 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:54.124974 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:20:55.051497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.051433 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" podStartSLOduration=3.051418093 podStartE2EDuration="3.051418093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:20:54.24951035 +0000 UTC m=+6477.337847021" watchObservedRunningTime="2026-11-01 10:20:55.051418093 +0000 UTC m=+6478.139754765" Nov 01 10:20:55.057519 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.057489 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk"] Nov 01 10:20:55.103973 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.103937 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tzrxk"] Nov 01 10:20:55.118261 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.118224 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p"] Nov 01 10:20:55.122469 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.122446 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.177641 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.177610 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p"] Nov 01 10:20:55.179473 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.179447 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:20:55.195306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.195254 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.195306 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.195306 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.195604 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.195337 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.195604 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.195408 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dtw\" (UniqueName: \"kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.226586 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.226554 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:20:55.226586 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.226574 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7zt8\"" Nov 01 10:20:55.226586 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.226554 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-23b415-serving-cert\"" Nov 01 10:20:55.226864 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.226653 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 10:20:55.296497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.296456 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.296497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.296497 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.296849 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.296528 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.296849 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.296547 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dtw\" (UniqueName: \"kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.296849 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:20:55.296631 5710 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-serving-cert: secret "maas-api-e2e-worker-single-main-23b415-serving-cert" not found Nov 01 10:20:55.296849 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:20:55.296749 5710 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls podName:23c0d92b-575d-43de-b926-c1a87458dbcc nodeName:}" failed. No retries permitted until 2026-11-01 10:20:55.796692089 +0000 UTC m=+6478.885028752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls") pod "maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" (UID: "23c0d92b-575d-43de-b926-c1a87458dbcc") : secret "maas-api-e2e-worker-single-main-23b415-serving-cert" not found Nov 01 10:20:55.297342 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.297320 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.299346 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.299322 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.361046 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.361009 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dtw\" (UniqueName: \"kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.510065 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.501471 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:20:55.812153 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.812049 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:55.814659 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:55.814631 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:56.037831 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:56.037784 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:56.421481 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:20:56.421449 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c0d92b_575d_43de_b926_c1a87458dbcc.slice/crio-76b81dc5e4e948f0cd409619adcbf92117cd2db1203e3a5f73cd517f7b529c36 WatchSource:0}: Error finding container 76b81dc5e4e948f0cd409619adcbf92117cd2db1203e3a5f73cd517f7b529c36: Status 404 returned error can't find the container with id 76b81dc5e4e948f0cd409619adcbf92117cd2db1203e3a5f73cd517f7b529c36 Nov 01 10:20:56.423315 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:56.423292 5710 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 10:20:56.423594 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:56.423572 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p"] Nov 01 10:20:57.057173 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:57.057137 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386bcb96-8863-4cae-9458-3a33f5969612" path="/var/lib/kubelet/pods/386bcb96-8863-4cae-9458-3a33f5969612/volumes" Nov 01 10:20:57.146156 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:57.146109 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" event={"ID":"23c0d92b-575d-43de-b926-c1a87458dbcc","Type":"ContainerStarted","Data":"76b81dc5e4e948f0cd409619adcbf92117cd2db1203e3a5f73cd517f7b529c36"} Nov 01 10:20:58.154078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:58.154037 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" event={"ID":"23c0d92b-575d-43de-b926-c1a87458dbcc","Type":"ContainerStarted","Data":"69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2"} Nov 01 10:20:58.154680 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:58.154157 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:20:58.310256 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:20:58.310202 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" podStartSLOduration=2.345173274 podStartE2EDuration="3.310185093s" totalImagesPullingTime="965.011819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:20:58.309744883 +0000 UTC m=+6481.398081550" watchObservedRunningTime="2026-11-01 10:20:58.310185093 +0000 UTC m=+6481.398521766" Nov 01 10:21:04.166636 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:04.166434 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:21:05.135731 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:05.135698 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:21:06.901752 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:06.900652 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:21:12.268247 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:12.268197 5710 generic.go:356] "Generic (PLEG): container finished" podID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerID="c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00" exitCode=0 Nov 01 10:21:12.268750 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:12.268260 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" event={"ID":"cc808c31-25df-476c-8c57-3cc1eb4a3b49","Type":"ContainerDied","Data":"c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00"} Nov 01 10:21:12.268750 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:12.268661 5710 scope.go:122] "RemoveContainer" containerID="c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00" Nov 01 10:21:12.270015 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:12.269983 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:21:12.270085 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:12.270064 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:21:12.270125 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:12.270109 5710 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 10:21:13.275797 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:13.275762 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" event={"ID":"cc808c31-25df-476c-8c57-3cc1eb4a3b49","Type":"ContainerStarted","Data":"8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be"} Nov 01 10:21:14.039679 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:14.039636 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:21:14.040633 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:14.040610 5710 patch_prober.go:29] interesting pod/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 10:21:14.040765 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:14.040686 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 10:21:15.039486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:15.039451 5710 patch_prober.go:29] interesting pod/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 10:21:15.039936 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:15.039516 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 10:21:15.480271 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:21:15.480231 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:21:16.039779 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:16.039696 5710 patch_prober.go:29] interesting pod/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 10:21:16.040311 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:16.039822 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 10:21:17.039840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:17.039806 5710 patch_prober.go:29] interesting pod/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 10:21:17.040293 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:17.039861 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 10:21:17.097745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:17.093836 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:21:18.038883 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:18.038847 5710 patch_prober.go:29] interesting pod/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 10:21:18.039087 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:18.038902 5710 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 10:21:19.043512 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:19.043465 5710 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:21:19.044782 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:19.044758 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:21:19.057794 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:19.057761 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794"] Nov 01 10:21:19.105096 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:19.105059 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8f794"] Nov 01 10:21:21.057296 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:21.057247 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3" path="/var/lib/kubelet/pods/3a3e85ef-7cd2-41a7-987a-fbc1ae4bbef3/volumes" Nov 01 10:21:21.778503 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:21.778457 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:21:31.057695 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:31.057660 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf"] Nov 01 10:21:31.105996 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:31.105959 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4h9pf"] Nov 01 10:21:33.057208 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:33.057160 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1161c0a3-231b-48c8-a2f4-8ffc26882f2b" path="/var/lib/kubelet/pods/1161c0a3-231b-48c8-a2f4-8ffc26882f2b/volumes" Nov 01 10:21:44.683121 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:21:44.683076 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:21:45.478900 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:21:45.478864 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:22:01.351514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:01.351478 5710 scope.go:122] "RemoveContainer" containerID="7ce0c3fa5b0bf4765e31354189857d141c1f5d608a2c31434e6a5e2698470aee" Nov 01 10:22:01.361386 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:01.361356 5710 scope.go:122] "RemoveContainer" containerID="815a3e99afb9e5a5916657631f3b0706e5486fb506863b9fad235fe4be34f1f9" Nov 01 10:22:06.051646 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:06.051607 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh"] Nov 01 10:22:06.103376 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:06.103334 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jtzmh"] Nov 01 10:22:07.057676 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:07.057640 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e56962a-0a15-4650-99dd-a3bceeee3ae6" path="/var/lib/kubelet/pods/0e56962a-0a15-4650-99dd-a3bceeee3ae6/volumes" Nov 01 10:22:14.070231 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:14.070192 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:22:15.482129 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:22:15.482084 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:22:17.057275 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:17.057244 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj"] Nov 01 10:22:17.103801 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:17.103748 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zljsj"] Nov 01 10:22:17.793174 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:17.793120 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p"] Nov 01 10:22:17.793174 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:17.793427 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" podUID="23c0d92b-575d-43de-b926-c1a87458dbcc" containerName="maas-api" containerID="cri-o://69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2" gracePeriod=30 Nov 01 10:22:18.256672 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:18.256628 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c"] Nov 01 10:22:19.044947 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.044918 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:22:19.057599 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.057562 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1c9053-af28-41e3-b608-69ce5329e3ad" path="/var/lib/kubelet/pods/9c1c9053-af28-41e3-b608-69ce5329e3ad/volumes" Nov 01 10:22:19.160842 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.160709 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\") pod \"23c0d92b-575d-43de-b926-c1a87458dbcc\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " Nov 01 10:22:19.160842 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.160815 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca\") pod \"23c0d92b-575d-43de-b926-c1a87458dbcc\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " Nov 01 10:22:19.161078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.160879 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw\" (UniqueName: \"kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw\") pod \"23c0d92b-575d-43de-b926-c1a87458dbcc\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " Nov 01 10:22:19.161078 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.160999 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs\") pod \"23c0d92b-575d-43de-b926-c1a87458dbcc\" (UID: \"23c0d92b-575d-43de-b926-c1a87458dbcc\") " Nov 01 10:22:19.161430 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.161382 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca" pod "23c0d92b-575d-43de-b926-c1a87458dbcc" (UID: "23c0d92b-575d-43de-b926-c1a87458dbcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:22:19.161877 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.161570 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d92b-575d-43de-b926-c1a87458dbcc-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:19.164000 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.163970 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs" pod "23c0d92b-575d-43de-b926-c1a87458dbcc" (UID: "23c0d92b-575d-43de-b926-c1a87458dbcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:22:19.164133 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.164080 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw" pod "23c0d92b-575d-43de-b926-c1a87458dbcc" (UID: "23c0d92b-575d-43de-b926-c1a87458dbcc"). InnerVolumeSpecName "kube-api-access-j2dtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:22:19.164323 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.164288 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls" pod "23c0d92b-575d-43de-b926-c1a87458dbcc" (UID: "23c0d92b-575d-43de-b926-c1a87458dbcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:22:19.262421 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.262389 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:19.262421 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.262417 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c0d92b-575d-43de-b926-c1a87458dbcc-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:19.262421 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.262428 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dtw\" (UniqueName: \"kubernetes.io/projected/23c0d92b-575d-43de-b926-c1a87458dbcc-kube-api-access-j2dtw\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:19.682589 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.682546 5710 generic.go:356] "Generic (PLEG): container finished" podID="23c0d92b-575d-43de-b926-c1a87458dbcc" containerID="69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2" exitCode=0 Nov 01 10:22:19.682810 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.682594 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" event={"ID":"23c0d92b-575d-43de-b926-c1a87458dbcc","Type":"ContainerDied","Data":"69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2"} Nov 01 10:22:19.682810 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.682604 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" Nov 01 10:22:19.682810 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.682648 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p" event={"ID":"23c0d92b-575d-43de-b926-c1a87458dbcc","Type":"ContainerDied","Data":"76b81dc5e4e948f0cd409619adcbf92117cd2db1203e3a5f73cd517f7b529c36"} Nov 01 10:22:19.682810 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.682679 5710 scope.go:122] "RemoveContainer" containerID="69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2" Nov 01 10:22:19.693817 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.693799 5710 scope.go:122] "RemoveContainer" containerID="69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2" Nov 01 10:22:19.694104 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:22:19.694083 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2\": container with ID starting with 69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2 not found: ID does not exist" containerID="69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2" Nov 01 10:22:19.694169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.694116 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2"} err="failed to get container status \"69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2\": rpc error: code = NotFound desc = could not find container \"69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2\": container with ID starting with 69f5ef4d81df59fedc7cd93e490dbd29569588657a51e55fe2505e07875cdae2 not found: ID does not exist" Nov 01 10:22:19.853843 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.853810 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p"] Nov 01 10:22:19.905811 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:19.905772 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-23b415-65548c8797-5m24p"] Nov 01 10:22:21.063795 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:21.063756 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c0d92b-575d-43de-b926-c1a87458dbcc" path="/var/lib/kubelet/pods/23c0d92b-575d-43de-b926-c1a87458dbcc/volumes" Nov 01 10:22:23.257726 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:23.257677 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerName="payload-processing" containerID="cri-o://188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd" gracePeriod=25 Nov 01 10:22:25.133827 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:25.133783 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:22:27.064341 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:27.064303 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:22:35.133655 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:35.133566 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 10:22:38.904307 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:38.904270 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2"] Nov 01 10:22:38.904865 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:38.904506 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" containerID="cri-o://8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be" gracePeriod=30 Nov 01 10:22:40.470620 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.470564 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz"] Nov 01 10:22:40.522109 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.522076 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qq5wz"] Nov 01 10:22:40.587511 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.587471 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh"] Nov 01 10:22:40.588046 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.588026 5710 cpu_manager.go:414] "containerMap: removing container" podUID="23c0d92b-575d-43de-b926-c1a87458dbcc" containerName="maas-api" Nov 01 10:22:40.588170 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.588048 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c0d92b-575d-43de-b926-c1a87458dbcc" containerName="maas-api" Nov 01 10:22:40.588233 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.588195 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c0d92b-575d-43de-b926-c1a87458dbcc" containerName="maas-api" Nov 01 10:22:40.592845 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.592817 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.646565 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.646528 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh"] Nov 01 10:22:40.647591 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.647558 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:22:40.647812 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.647564 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:22:40.647812 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.647570 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:22:40.647919 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.647674 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:22:40.695227 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.695183 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.695465 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.695310 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.695465 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.695361 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgwq\" (UniqueName: \"kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.796938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.796847 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.796938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.796913 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.797170 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.796948 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgwq\" (UniqueName: \"kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.797655 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.797631 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.799561 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.799535 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.861169 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.861137 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgwq\" (UniqueName: \"kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq\") pod \"maas-api-revoke-sub-3d4156f1-xpdxh\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:40.906834 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:40.906790 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:41.057581 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:41.057492 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064c8670-1d0f-47db-a91c-d0651cb27789" path="/var/lib/kubelet/pods/064c8670-1d0f-47db-a91c-d0651cb27789/volumes" Nov 01 10:22:41.282841 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:22:41.282795 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15f2861_719f_401e_a6d4_6e23ea760b8e.slice/crio-03b247a37c4e49260ac37db19b226290b7731a5a2fde3cc6c1ac3c3c24080e60 WatchSource:0}: Error finding container 03b247a37c4e49260ac37db19b226290b7731a5a2fde3cc6c1ac3c3c24080e60: Status 404 returned error can't find the container with id 03b247a37c4e49260ac37db19b226290b7731a5a2fde3cc6c1ac3c3c24080e60 Nov 01 10:22:41.284024 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:41.283999 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh"] Nov 01 10:22:41.815802 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:41.815766 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" event={"ID":"f15f2861-719f-401e-a6d4-6e23ea760b8e","Type":"ContainerStarted","Data":"bfaab4c85196cd4b7119fa355193b2ef8547bc9774e2c2226d6c2f3544ae462b"} Nov 01 10:22:41.816345 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:41.815839 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" event={"ID":"f15f2861-719f-401e-a6d4-6e23ea760b8e","Type":"ContainerStarted","Data":"03b247a37c4e49260ac37db19b226290b7731a5a2fde3cc6c1ac3c3c24080e60"} Nov 01 10:22:41.939578 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:41.939468 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" podStartSLOduration=1.939453779 podStartE2EDuration="1.939453779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:22:41.938307829 +0000 UTC m=+6585.026644501" watchObservedRunningTime="2026-11-01 10:22:41.939453779 +0000 UTC m=+6585.027790486" Nov 01 10:22:42.823345 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:42.823309 5710 generic.go:356] "Generic (PLEG): container finished" podID="f15f2861-719f-401e-a6d4-6e23ea760b8e" containerID="bfaab4c85196cd4b7119fa355193b2ef8547bc9774e2c2226d6c2f3544ae462b" exitCode=0 Nov 01 10:22:42.823872 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:42.823416 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" event={"ID":"f15f2861-719f-401e-a6d4-6e23ea760b8e","Type":"ContainerDied","Data":"bfaab4c85196cd4b7119fa355193b2ef8547bc9774e2c2226d6c2f3544ae462b"} Nov 01 10:22:43.512763 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.512737 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:22:43.626334 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.626299 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr\" (UniqueName: \"kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr\") pod \"3046a42c-fb23-41f3-9ee2-56466bab3a21\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " Nov 01 10:22:43.626552 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.626415 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume\") pod \"3046a42c-fb23-41f3-9ee2-56466bab3a21\" (UID: \"3046a42c-fb23-41f3-9ee2-56466bab3a21\") " Nov 01 10:22:43.626848 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.626823 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume" pod "3046a42c-fb23-41f3-9ee2-56466bab3a21" (UID: "3046a42c-fb23-41f3-9ee2-56466bab3a21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:22:43.629241 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.629198 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr" pod "3046a42c-fb23-41f3-9ee2-56466bab3a21" (UID: "3046a42c-fb23-41f3-9ee2-56466bab3a21"). InnerVolumeSpecName "kube-api-access-hvphr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:22:43.727804 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.727766 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvphr\" (UniqueName: \"kubernetes.io/projected/3046a42c-fb23-41f3-9ee2-56466bab3a21-kube-api-access-hvphr\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:43.727804 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.727800 5710 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046a42c-fb23-41f3-9ee2-56466bab3a21-plugins-config-volume\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:43.830330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.830295 5710 generic.go:356] "Generic (PLEG): container finished" podID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerID="188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd" exitCode=0 Nov 01 10:22:43.830883 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.830360 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" Nov 01 10:22:43.830883 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.830362 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" event={"ID":"3046a42c-fb23-41f3-9ee2-56466bab3a21","Type":"ContainerDied","Data":"188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd"} Nov 01 10:22:43.830883 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.830464 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c" event={"ID":"3046a42c-fb23-41f3-9ee2-56466bab3a21","Type":"ContainerDied","Data":"07526fae02df6428b231ebca265fbe6dd1dc7914562625899d3153fbae280089"} Nov 01 10:22:43.830883 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.830514 5710 scope.go:122] "RemoveContainer" containerID="188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd" Nov 01 10:22:43.840834 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.840806 5710 scope.go:122] "RemoveContainer" containerID="188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd" Nov 01 10:22:43.841148 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:22:43.841128 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd\": container with ID starting with 188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd not found: ID does not exist" containerID="188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd" Nov 01 10:22:43.841219 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:43.841159 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd"} err="failed to get container status \"188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd\": rpc error: code = NotFound desc = could not find container \"188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd\": container with ID starting with 188a42259d963ee99c72e382d87225feb9195f7d91d70a04a6117a6611eaf1cd not found: ID does not exist" Nov 01 10:22:44.001861 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.001826 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c"] Nov 01 10:22:44.007595 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.007566 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-23b415-864c54b79pn77c"] Nov 01 10:22:44.017957 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.017928 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:44.132354 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.132320 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca\") pod \"f15f2861-719f-401e-a6d4-6e23ea760b8e\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " Nov 01 10:22:44.132545 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.132386 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca\") pod \"f15f2861-719f-401e-a6d4-6e23ea760b8e\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " Nov 01 10:22:44.132545 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.132532 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq\" (UniqueName: \"kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq\") pod \"f15f2861-719f-401e-a6d4-6e23ea760b8e\" (UID: \"f15f2861-719f-401e-a6d4-6e23ea760b8e\") " Nov 01 10:22:44.132804 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.132775 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca" pod "f15f2861-719f-401e-a6d4-6e23ea760b8e" (UID: "f15f2861-719f-401e-a6d4-6e23ea760b8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:22:44.132923 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.132902 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.135478 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.135443 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq" pod "f15f2861-719f-401e-a6d4-6e23ea760b8e" (UID: "f15f2861-719f-401e-a6d4-6e23ea760b8e"). InnerVolumeSpecName "kube-api-access-4zgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:22:44.135857 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.135834 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca" pod "f15f2861-719f-401e-a6d4-6e23ea760b8e" (UID: "f15f2861-719f-401e-a6d4-6e23ea760b8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:22:44.154844 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.154821 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:22:44.234156 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.234117 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zgwq\" (UniqueName: \"kubernetes.io/projected/f15f2861-719f-401e-a6d4-6e23ea760b8e-kube-api-access-4zgwq\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.234156 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.234150 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f2861-719f-401e-a6d4-6e23ea760b8e-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.335339 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335299 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335361 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335458 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335528 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335485 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335775 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335525 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335775 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335565 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335775 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335655 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.335775 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335697 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.336018 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335822 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:22:44.336018 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335835 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:22:44.336018 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.335899 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert\") pod \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\" (UID: \"cc808c31-25df-476c-8c57-3cc1eb4a3b49\") " Nov 01 10:22:44.336018 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336010 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:22:44.336262 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336086 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:22:44.336420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336367 5710 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-credential-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.336420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336383 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:22:44.336420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336396 5710 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-data\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.336420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336410 5710 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.336420 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.336424 5710 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-workload-socket\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.338112 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.338086 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "kube-api-access-8gld5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:22:44.338219 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.338121 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 10:22:44.338398 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.338370 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:22:44.338445 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.338388 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo" pod "cc808c31-25df-476c-8c57-3cc1eb4a3b49" (UID: "cc808c31-25df-476c-8c57-3cc1eb4a3b49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 10:22:44.437525 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.437423 5710 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-podinfo\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.437525 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.437462 5710 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-envoy\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.437525 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.437474 5710 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istio-token\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.437525 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.437489 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gld5\" (UniqueName: \"kubernetes.io/projected/cc808c31-25df-476c-8c57-3cc1eb4a3b49-kube-api-access-8gld5\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.437525 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.437502 5710 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc808c31-25df-476c-8c57-3cc1eb4a3b49-istiod-ca-cert\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:22:44.840779 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.840739 5710 generic.go:356] "Generic (PLEG): container finished" podID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerID="8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be" exitCode=0 Nov 01 10:22:44.841303 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.840804 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" Nov 01 10:22:44.841303 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.840834 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" event={"ID":"cc808c31-25df-476c-8c57-3cc1eb4a3b49","Type":"ContainerDied","Data":"8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be"} Nov 01 10:22:44.841303 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.840906 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2" event={"ID":"cc808c31-25df-476c-8c57-3cc1eb4a3b49","Type":"ContainerDied","Data":"33efe6ba8e2d8abab80b7e8d8075ccd17cece7b2c1bd1126dbeb5cc6c908d2ad"} Nov 01 10:22:44.841303 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.840922 5710 scope.go:122] "RemoveContainer" containerID="8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be" Nov 01 10:22:44.842458 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.842314 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" Nov 01 10:22:44.842458 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.842330 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh" event={"ID":"f15f2861-719f-401e-a6d4-6e23ea760b8e","Type":"ContainerDied","Data":"03b247a37c4e49260ac37db19b226290b7731a5a2fde3cc6c1ac3c3c24080e60"} Nov 01 10:22:44.842458 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.842378 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b247a37c4e49260ac37db19b226290b7731a5a2fde3cc6c1ac3c3c24080e60" Nov 01 10:22:44.851533 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.851448 5710 scope.go:122] "RemoveContainer" containerID="c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00" Nov 01 10:22:44.861581 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.861556 5710 scope.go:122] "RemoveContainer" containerID="8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be" Nov 01 10:22:44.861944 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:22:44.861925 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be\": container with ID starting with 8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be not found: ID does not exist" containerID="8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be" Nov 01 10:22:44.862022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.861955 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be"} err="failed to get container status \"8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be\": rpc error: code = NotFound desc = could not find container \"8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be\": container with ID starting with 8853458a8951f6460f65fd4e9d03e9a49f6fb6328742db1441abe868bc4327be not found: ID does not exist" Nov 01 10:22:44.862022 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.861978 5710 scope.go:122] "RemoveContainer" containerID="c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00" Nov 01 10:22:44.862237 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:22:44.862219 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00\": container with ID starting with c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00 not found: ID does not exist" containerID="c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00" Nov 01 10:22:44.862298 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:44.862249 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00"} err="failed to get container status \"c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00\": rpc error: code = NotFound desc = could not find container \"c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00\": container with ID starting with c60034264df225bfc99cad1cd01314bc46aa36714cbb8004d7e5fefe6f348f00 not found: ID does not exist" Nov 01 10:22:45.012052 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:45.012015 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2"] Nov 01 10:22:45.027268 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:45.027237 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-23b415-openshift-default-6bd6df6877fq5n2"] Nov 01 10:22:45.057141 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:45.057110 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" path="/var/lib/kubelet/pods/3046a42c-fb23-41f3-9ee2-56466bab3a21/volumes" Nov 01 10:22:45.057551 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:45.057538 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" path="/var/lib/kubelet/pods/cc808c31-25df-476c-8c57-3cc1eb4a3b49/volumes" Nov 01 10:22:45.478091 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:22:45.478053 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:22:52.057117 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:52.057079 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx"] Nov 01 10:22:52.108943 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:52.108904 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tp5mx"] Nov 01 10:22:53.057743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:53.057669 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4b1a75-734f-43b2-bc9a-b70a963ec21e" path="/var/lib/kubelet/pods/0b4b1a75-734f-43b2-bc9a-b70a963ec21e/volumes" Nov 01 10:22:55.071920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:55.071886 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:22:57.664889 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:57.664846 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:22:57.665347 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:22:57.664939 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:23:01.389574 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:01.389529 5710 scope.go:122] "RemoveContainer" containerID="2e29f4927a436d6a3687e232618184592355825bfb1b80276661dbb7a3307ac7" Nov 01 10:23:01.399035 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:01.399013 5710 scope.go:122] "RemoveContainer" containerID="7101f97abfc02595dcb44c6a0e672968adaf00b7f81e297cd5c0c5aaab17e92c" Nov 01 10:23:01.560326 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:01.560290 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:23:01.568974 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:01.568951 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:23:12.051866 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:12.051821 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5"] Nov 01 10:23:12.103599 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:12.103551 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-h25z5"] Nov 01 10:23:13.057863 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:13.057816 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7d4e8b-3781-4e59-a4b6-7a1dabaec911" path="/var/lib/kubelet/pods/da7d4e8b-3781-4e59-a4b6-7a1dabaec911/volumes" Nov 01 10:23:15.477119 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:23:15.477075 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:23:32.062244 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:32.062205 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb"] Nov 01 10:23:32.113602 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:32.113569 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rz8cb"] Nov 01 10:23:33.057782 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:33.057747 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6492bfa0-a381-4dea-a722-317238b0f5d2" path="/var/lib/kubelet/pods/6492bfa0-a381-4dea-a722-317238b0f5d2/volumes" Nov 01 10:23:45.476823 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:23:45.476782 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:23:50.054016 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:50.053977 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz"] Nov 01 10:23:50.105409 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:50.105371 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8bdtz"] Nov 01 10:23:51.057060 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:23:51.057025 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685651e8-112b-4121-a95a-e8ec5d5e9557" path="/var/lib/kubelet/pods/685651e8-112b-4121-a95a-e8ec5d5e9557/volumes" Nov 01 10:24:01.436824 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:01.436788 5710 scope.go:122] "RemoveContainer" containerID="e46ca2732543f2da77db2e8c8ae780c552b9f25cd5db0148631093757caf37a6" Nov 01 10:24:01.446430 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:01.446409 5710 scope.go:122] "RemoveContainer" containerID="46c2cd74eecbaa436a70bba10bb7c4d8d93877a9c632789c6964b1de03dbfd98" Nov 01 10:24:01.455458 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:01.455438 5710 scope.go:122] "RemoveContainer" containerID="4f78e6de7554abea27ce249c5a69b1c4861ead222cf96d11375679fa0ac9eb5c" Nov 01 10:24:15.477739 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:24:15.477681 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:24:27.660910 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:27.660861 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:24:27.661710 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:27.661645 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:24:29.790468 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:29.790428 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:24:32.311760 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.311708 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz"] Nov 01 10:24:32.312189 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312172 5710 cpu_manager.go:414] "containerMap: removing container" podUID="f15f2861-719f-401e-a6d4-6e23ea760b8e" containerName="revoke-keys" Nov 01 10:24:32.312189 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312183 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15f2861-719f-401e-a6d4-6e23ea760b8e" containerName="revoke-keys" Nov 01 10:24:32.312276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312192 5710 cpu_manager.go:414] "containerMap: removing container" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerName="payload-processing" Nov 01 10:24:32.312276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312197 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerName="payload-processing" Nov 01 10:24:32.312276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312216 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" Nov 01 10:24:32.312276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312223 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" Nov 01 10:24:32.312276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312235 5710 cpu_manager.go:414] "containerMap: removing container" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" Nov 01 10:24:32.312276 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312240 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" Nov 01 10:24:32.312483 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312301 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15f2861-719f-401e-a6d4-6e23ea760b8e" containerName="revoke-keys" Nov 01 10:24:32.312483 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312310 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="3046a42c-fb23-41f3-9ee2-56466bab3a21" containerName="payload-processing" Nov 01 10:24:32.312483 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312321 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" Nov 01 10:24:32.312483 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.312327 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc808c31-25df-476c-8c57-3cc1eb4a3b49" containerName="istio-proxy" Nov 01 10:24:32.315049 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.315031 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.370875 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.370844 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz"] Nov 01 10:24:32.414747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.414676 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hrb\" (UniqueName: \"kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.414747 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.414746 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.414981 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.414956 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.420583 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.420558 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:24:32.420675 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.420564 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:24:32.421508 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.421493 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:24:32.421818 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.421799 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:24:32.516456 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.516407 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.516456 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.516455 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hrb\" (UniqueName: \"kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.516804 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.516565 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.517133 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.517107 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.519144 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.519123 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.580850 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.580773 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hrb\" (UniqueName: \"kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb\") pod \"maas-api-revoke-sub-d576e68b-97xxz\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:32.627585 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:32.627543 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:33.001597 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:24:33.001555 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43582e8_aeaf_4bbe_b964_3255a9e2b38b.slice/crio-20d3d8285d1978b118bbc2b70eb7c755355ab8ea8fbd4293eea173d92ebdf847 WatchSource:0}: Error finding container 20d3d8285d1978b118bbc2b70eb7c755355ab8ea8fbd4293eea173d92ebdf847: Status 404 returned error can't find the container with id 20d3d8285d1978b118bbc2b70eb7c755355ab8ea8fbd4293eea173d92ebdf847 Nov 01 10:24:33.003617 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:33.003590 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz"] Nov 01 10:24:33.495732 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:33.495687 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" event={"ID":"e43582e8-aeaf-4bbe-b964-3255a9e2b38b","Type":"ContainerStarted","Data":"cfbf02c3dfcc27a68e17c1d51eb64bf59bc00bd86ef13f86208e6f2330e3ac1a"} Nov 01 10:24:33.496200 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:33.495813 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" event={"ID":"e43582e8-aeaf-4bbe-b964-3255a9e2b38b","Type":"ContainerStarted","Data":"20d3d8285d1978b118bbc2b70eb7c755355ab8ea8fbd4293eea173d92ebdf847"} Nov 01 10:24:33.617943 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:33.617878 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" podStartSLOduration=1.6178645729999999 podStartE2EDuration="1.617864573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:24:33.616317436 +0000 UTC m=+6696.704654131" watchObservedRunningTime="2026-11-01 10:24:33.617864573 +0000 UTC m=+6696.706201243" Nov 01 10:24:34.504048 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:34.504010 5710 generic.go:356] "Generic (PLEG): container finished" podID="e43582e8-aeaf-4bbe-b964-3255a9e2b38b" containerID="cfbf02c3dfcc27a68e17c1d51eb64bf59bc00bd86ef13f86208e6f2330e3ac1a" exitCode=0 Nov 01 10:24:34.504482 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:34.504076 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" event={"ID":"e43582e8-aeaf-4bbe-b964-3255a9e2b38b","Type":"ContainerDied","Data":"cfbf02c3dfcc27a68e17c1d51eb64bf59bc00bd86ef13f86208e6f2330e3ac1a"} Nov 01 10:24:35.652154 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.652129 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:35.752311 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.752276 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb\" (UniqueName: \"kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb\") pod \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " Nov 01 10:24:35.752569 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.752346 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca\") pod \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " Nov 01 10:24:35.752569 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.752400 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca\") pod \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\" (UID: \"e43582e8-aeaf-4bbe-b964-3255a9e2b38b\") " Nov 01 10:24:35.752823 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.752799 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca" pod "e43582e8-aeaf-4bbe-b964-3255a9e2b38b" (UID: "e43582e8-aeaf-4bbe-b964-3255a9e2b38b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:24:35.754607 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.754586 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca" pod "e43582e8-aeaf-4bbe-b964-3255a9e2b38b" (UID: "e43582e8-aeaf-4bbe-b964-3255a9e2b38b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:24:35.754702 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.754682 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb" pod "e43582e8-aeaf-4bbe-b964-3255a9e2b38b" (UID: "e43582e8-aeaf-4bbe-b964-3255a9e2b38b"). InnerVolumeSpecName "kube-api-access-g6hrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:24:35.853595 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.853554 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:24:35.853595 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.853591 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:24:35.853595 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:35.853601 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hrb\" (UniqueName: \"kubernetes.io/projected/e43582e8-aeaf-4bbe-b964-3255a9e2b38b-kube-api-access-g6hrb\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:24:36.516206 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:36.516167 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" event={"ID":"e43582e8-aeaf-4bbe-b964-3255a9e2b38b","Type":"ContainerDied","Data":"20d3d8285d1978b118bbc2b70eb7c755355ab8ea8fbd4293eea173d92ebdf847"} Nov 01 10:24:36.516448 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:36.516226 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d3d8285d1978b118bbc2b70eb7c755355ab8ea8fbd4293eea173d92ebdf847" Nov 01 10:24:36.516448 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:36.516185 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz" Nov 01 10:24:45.183600 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:45.183556 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:24:45.503969 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:24:45.503866 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:24:51.938745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:51.923663 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7"] Nov 01 10:24:51.938745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:51.928205 5710 cpu_manager.go:414] "containerMap: removing container" podUID="e43582e8-aeaf-4bbe-b964-3255a9e2b38b" containerName="revoke-keys" Nov 01 10:24:51.938745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:51.928236 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43582e8-aeaf-4bbe-b964-3255a9e2b38b" containerName="revoke-keys" Nov 01 10:24:51.938745 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:51.928402 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43582e8-aeaf-4bbe-b964-3255a9e2b38b" containerName="revoke-keys" Nov 01 10:24:51.946756 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:51.945076 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:51.988753 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:51.987852 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7"] Nov 01 10:24:52.048746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.047761 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.048746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.047907 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.048746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.047981 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdj7h\" (UniqueName: \"kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.063031 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.062992 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:24:52.063396 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.063374 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 10:24:52.063689 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.063670 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:24:52.063965 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.063948 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:24:52.160601 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.160560 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.160996 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.160971 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdj7h\" (UniqueName: \"kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.161287 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.161253 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.161454 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.161426 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.164495 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.164466 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.247810 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.245684 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdj7h\" (UniqueName: \"kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h\") pod \"maas-api-revoke-sub-d2bc93bc-vl6s7\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.269243 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.269204 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:52.773873 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:52.773843 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7"] Nov 01 10:24:52.778059 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:24:52.778008 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d827009_d262_40f7_93c0_0b1c68039f40.slice/crio-41d5e49314e6194f5b85e36945e5a4e43ab1b52472383bc93847df751540679d WatchSource:0}: Error finding container 41d5e49314e6194f5b85e36945e5a4e43ab1b52472383bc93847df751540679d: Status 404 returned error can't find the container with id 41d5e49314e6194f5b85e36945e5a4e43ab1b52472383bc93847df751540679d Nov 01 10:24:53.737225 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:53.737164 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" event={"ID":"6d827009-d262-40f7-93c0-0b1c68039f40","Type":"ContainerStarted","Data":"cbfc04a4ef1bd08c2edc0d1a3d8c1b2dcdea1851963b051c4a622b4d6d7cea3d"} Nov 01 10:24:53.739068 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:53.739033 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" event={"ID":"6d827009-d262-40f7-93c0-0b1c68039f40","Type":"ContainerStarted","Data":"41d5e49314e6194f5b85e36945e5a4e43ab1b52472383bc93847df751540679d"} Nov 01 10:24:53.863609 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:53.863559 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" podStartSLOduration=2.863541298 podStartE2EDuration="2.863541298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:24:53.861669139 +0000 UTC m=+6716.950005804" watchObservedRunningTime="2026-11-01 10:24:53.863541298 +0000 UTC m=+6716.951877971" Nov 01 10:24:54.353975 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:54.353927 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:24:54.354225 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:54.353990 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:24:54.744707 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:54.744618 5710 generic.go:356] "Generic (PLEG): container finished" podID="6d827009-d262-40f7-93c0-0b1c68039f40" containerID="cbfc04a4ef1bd08c2edc0d1a3d8c1b2dcdea1851963b051c4a622b4d6d7cea3d" exitCode=0 Nov 01 10:24:54.745115 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:54.744699 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" event={"ID":"6d827009-d262-40f7-93c0-0b1c68039f40","Type":"ContainerDied","Data":"cbfc04a4ef1bd08c2edc0d1a3d8c1b2dcdea1851963b051c4a622b4d6d7cea3d"} Nov 01 10:24:55.285166 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:55.285118 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:24:55.890333 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:55.890309 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:56.030036 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.029945 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h\" (UniqueName: \"kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h\") pod \"6d827009-d262-40f7-93c0-0b1c68039f40\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " Nov 01 10:24:56.030036 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.030011 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca\") pod \"6d827009-d262-40f7-93c0-0b1c68039f40\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " Nov 01 10:24:56.030304 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.030136 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca\") pod \"6d827009-d262-40f7-93c0-0b1c68039f40\" (UID: \"6d827009-d262-40f7-93c0-0b1c68039f40\") " Nov 01 10:24:56.030515 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.030484 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca" pod "6d827009-d262-40f7-93c0-0b1c68039f40" (UID: "6d827009-d262-40f7-93c0-0b1c68039f40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:24:56.032434 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.032401 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca" pod "6d827009-d262-40f7-93c0-0b1c68039f40" (UID: "6d827009-d262-40f7-93c0-0b1c68039f40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:24:56.032551 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.032402 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h" pod "6d827009-d262-40f7-93c0-0b1c68039f40" (UID: "6d827009-d262-40f7-93c0-0b1c68039f40"). InnerVolumeSpecName "kube-api-access-mdj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:24:56.131289 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.131206 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-openshift-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:24:56.131289 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.131283 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdj7h\" (UniqueName: \"kubernetes.io/projected/6d827009-d262-40f7-93c0-0b1c68039f40-kube-api-access-mdj7h\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:24:56.131289 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.131294 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d827009-d262-40f7-93c0-0b1c68039f40-maas-api-xks-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:24:56.757302 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.757273 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" Nov 01 10:24:56.757514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.757268 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7" event={"ID":"6d827009-d262-40f7-93c0-0b1c68039f40","Type":"ContainerDied","Data":"41d5e49314e6194f5b85e36945e5a4e43ab1b52472383bc93847df751540679d"} Nov 01 10:24:56.757514 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:24:56.757410 5710 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41d5e49314e6194f5b85e36945e5a4e43ab1b52472383bc93847df751540679d" Nov 01 10:25:01.485265 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:01.485233 5710 scope.go:122] "RemoveContainer" containerID="0c4ac8cf94339a5de4e1cc488904ea978322e68f6924715dd056a588cf09c73d" Nov 01 10:25:01.494185 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:01.494167 5710 scope.go:122] "RemoveContainer" containerID="287f4e264a0e06a0123ea6e7c92cc9c277f74107105a0a250f0ad4d6b1daad1e" Nov 01 10:25:01.502335 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:01.502314 5710 scope.go:122] "RemoveContainer" containerID="2966ed217ea9781dfdbe3d96408d8a302cb134be875134f2205af1876cd945b4" Nov 01 10:25:01.510606 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:01.510581 5710 scope.go:122] "RemoveContainer" containerID="866abf4e5c8b060a8ad35d1382c6a401f2826f9ed5dfddba351ffeea7fc31d56" Nov 01 10:25:03.857449 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:03.857413 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:25:04.053754 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.053702 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb"] Nov 01 10:25:04.105337 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.105299 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-mx7sb"] Nov 01 10:25:04.887642 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.887605 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv"] Nov 01 10:25:04.888100 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.888065 5710 cpu_manager.go:414] "containerMap: removing container" podUID="6d827009-d262-40f7-93c0-0b1c68039f40" containerName="revoke-keys" Nov 01 10:25:04.888100 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.888078 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d827009-d262-40f7-93c0-0b1c68039f40" containerName="revoke-keys" Nov 01 10:25:04.888181 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.888150 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d827009-d262-40f7-93c0-0b1c68039f40" containerName="revoke-keys" Nov 01 10:25:04.893298 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.893276 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:04.908676 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.908647 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6"] Nov 01 10:25:04.912411 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.912387 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:04.948194 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.948160 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv"] Nov 01 10:25:04.959007 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.958980 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 10:25:04.998385 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.998343 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 10:25:04.998596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.998395 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 10:25:04.998596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.998436 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 10:25:04.998596 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:04.998541 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7zt8\"" Nov 01 10:25:05.001636 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.001612 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6"] Nov 01 10:25:05.021460 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021425 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-maas-api-tls\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.021460 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021462 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dv6x\" (UniqueName: \"kubernetes.io/projected/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-kube-api-access-2dv6x\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.021699 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021485 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-openshift-service-ca\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.021699 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021610 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxjk\" (UniqueName: \"kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.021699 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021692 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.021874 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021713 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.021874 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021833 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-metrics-certs\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.021977 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.021887 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.057375 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.057335 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ca5489-6277-4974-8387-0194782e21ad" path="/var/lib/kubelet/pods/b3ca5489-6277-4974-8387-0194782e21ad/volumes" Nov 01 10:25:05.122620 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122577 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.122620 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122626 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.122938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122650 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-metrics-certs\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.122938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122680 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.122938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122753 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-maas-api-tls\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.122938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122777 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dv6x\" (UniqueName: \"kubernetes.io/projected/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-kube-api-access-2dv6x\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.122938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122805 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-openshift-service-ca\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.122938 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.122873 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxjk\" (UniqueName: \"kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.123599 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.123572 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.123855 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.123834 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-openshift-service-ca\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.126049 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.126029 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-maas-api-tls\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.126049 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.126036 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-metrics-certs\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.126186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.126036 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.126186 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.126106 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.186922 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.186846 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxjk\" (UniqueName: \"kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk\") pod \"maas-api-7dcf475b5f-pfqsv\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.187068 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.186941 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dv6x\" (UniqueName: \"kubernetes.io/projected/c0558595-c99f-4be6-8ff8-5d9cfc349a9e-kube-api-access-2dv6x\") pod \"maas-api-676b7bff79-6h7t6\" (UID: \"c0558595-c99f-4be6-8ff8-5d9cfc349a9e\") " pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.207787 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.207753 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:05.225920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.225863 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:05.616014 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:25:05.615983 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c63098_43de_4ccc_a9d7_57623b19a384.slice/crio-c2847d050fe50e02ed6af0d513fccc29c41749dea295600d75c1250f6e7032b8 WatchSource:0}: Error finding container c2847d050fe50e02ed6af0d513fccc29c41749dea295600d75c1250f6e7032b8: Status 404 returned error can't find the container with id c2847d050fe50e02ed6af0d513fccc29c41749dea295600d75c1250f6e7032b8 Nov 01 10:25:05.618275 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.618143 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv"] Nov 01 10:25:05.641666 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:25:05.641630 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0558595_c99f_4be6_8ff8_5d9cfc349a9e.slice/crio-91affd05084114ebfc6fbfcef6e026cbce74db29db1b6a8cb912513fab9de5e8 WatchSource:0}: Error finding container 91affd05084114ebfc6fbfcef6e026cbce74db29db1b6a8cb912513fab9de5e8: Status 404 returned error can't find the container with id 91affd05084114ebfc6fbfcef6e026cbce74db29db1b6a8cb912513fab9de5e8 Nov 01 10:25:05.642863 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.642838 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6"] Nov 01 10:25:05.811783 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.811738 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" event={"ID":"c0558595-c99f-4be6-8ff8-5d9cfc349a9e","Type":"ContainerStarted","Data":"91affd05084114ebfc6fbfcef6e026cbce74db29db1b6a8cb912513fab9de5e8"} Nov 01 10:25:05.813055 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:05.813027 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" event={"ID":"d4c63098-43de-4ccc-a9d7-57623b19a384","Type":"ContainerStarted","Data":"c2847d050fe50e02ed6af0d513fccc29c41749dea295600d75c1250f6e7032b8"} Nov 01 10:25:06.820020 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:06.819920 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" event={"ID":"d4c63098-43de-4ccc-a9d7-57623b19a384","Type":"ContainerStarted","Data":"0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0"} Nov 01 10:25:06.820513 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:06.820092 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:06.825772 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:06.822486 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" event={"ID":"c0558595-c99f-4be6-8ff8-5d9cfc349a9e","Type":"ContainerStarted","Data":"6c13eacd843c39ddf5993e38eb5625c0c4be6290ae28c006dcc9571261aa75cf"} Nov 01 10:25:06.825772 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:06.823470 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:06.947126 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:06.947074 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" podStartSLOduration=2.025729661 podStartE2EDuration="2.947058823s" totalImagesPullingTime="921.329162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:25:06.945011488 +0000 UTC m=+6730.033348159" watchObservedRunningTime="2026-11-01 10:25:06.947058823 +0000 UTC m=+6730.035395494" Nov 01 10:25:07.096678 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:07.096617 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" podStartSLOduration=2.179068251 podStartE2EDuration="3.096601262s" totalImagesPullingTime="917.533011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:25:07.096031486 +0000 UTC m=+6730.184368156" watchObservedRunningTime="2026-11-01 10:25:07.096601262 +0000 UTC m=+6730.184937978" Nov 01 10:25:12.834728 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:12.834678 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:13.839912 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:13.839877 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-676b7bff79-6h7t6" Nov 01 10:25:14.017009 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.016968 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv"] Nov 01 10:25:14.017277 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.017240 5710 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" podUID="d4c63098-43de-4ccc-a9d7-57623b19a384" containerName="maas-api" containerID="cri-o://0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0" gracePeriod=30 Nov 01 10:25:14.264660 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.264634 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:14.321587 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.321551 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk\" (UniqueName: \"kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk\") pod \"d4c63098-43de-4ccc-a9d7-57623b19a384\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " Nov 01 10:25:14.321788 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.321616 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs\") pod \"d4c63098-43de-4ccc-a9d7-57623b19a384\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " Nov 01 10:25:14.321788 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.321655 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca\") pod \"d4c63098-43de-4ccc-a9d7-57623b19a384\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " Nov 01 10:25:14.321788 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.321701 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls\") pod \"d4c63098-43de-4ccc-a9d7-57623b19a384\" (UID: \"d4c63098-43de-4ccc-a9d7-57623b19a384\") " Nov 01 10:25:14.322063 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.322037 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca" pod "d4c63098-43de-4ccc-a9d7-57623b19a384" (UID: "d4c63098-43de-4ccc-a9d7-57623b19a384"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:25:14.322156 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.322141 5710 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c63098-43de-4ccc-a9d7-57623b19a384-openshift-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:25:14.323995 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.323965 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs" pod "d4c63098-43de-4ccc-a9d7-57623b19a384" (UID: "d4c63098-43de-4ccc-a9d7-57623b19a384"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:25:14.324096 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.324005 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls" pod "d4c63098-43de-4ccc-a9d7-57623b19a384" (UID: "d4c63098-43de-4ccc-a9d7-57623b19a384"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 10:25:14.324096 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.324049 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk" pod "d4c63098-43de-4ccc-a9d7-57623b19a384" (UID: "d4c63098-43de-4ccc-a9d7-57623b19a384"). InnerVolumeSpecName "kube-api-access-kjxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:25:14.423094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.422998 5710 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-metrics-certs\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:25:14.423094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.423042 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c63098-43de-4ccc-a9d7-57623b19a384-maas-api-tls\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:25:14.423094 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.423056 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxjk\" (UniqueName: \"kubernetes.io/projected/d4c63098-43de-4ccc-a9d7-57623b19a384-kube-api-access-kjxjk\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:25:14.865495 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.865451 5710 generic.go:356] "Generic (PLEG): container finished" podID="d4c63098-43de-4ccc-a9d7-57623b19a384" containerID="0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0" exitCode=0 Nov 01 10:25:14.865987 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.865514 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" Nov 01 10:25:14.865987 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.865538 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" event={"ID":"d4c63098-43de-4ccc-a9d7-57623b19a384","Type":"ContainerDied","Data":"0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0"} Nov 01 10:25:14.865987 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.865613 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv" event={"ID":"d4c63098-43de-4ccc-a9d7-57623b19a384","Type":"ContainerDied","Data":"c2847d050fe50e02ed6af0d513fccc29c41749dea295600d75c1250f6e7032b8"} Nov 01 10:25:14.865987 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.865636 5710 scope.go:122] "RemoveContainer" containerID="0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0" Nov 01 10:25:14.876330 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.876311 5710 scope.go:122] "RemoveContainer" containerID="0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0" Nov 01 10:25:14.876686 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:25:14.876661 5710 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0\": container with ID starting with 0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0 not found: ID does not exist" containerID="0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0" Nov 01 10:25:14.876822 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:14.876692 5710 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0"} err="failed to get container status \"0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0\": rpc error: code = NotFound desc = could not find container \"0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0\": container with ID starting with 0642d6b96028333cfbc6793ba8ac52d8c00d6d6d700145065f3e8a7b301305d0 not found: ID does not exist" Nov 01 10:25:15.037785 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:15.037742 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv"] Nov 01 10:25:15.089944 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:15.089910 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dcf475b5f-pfqsv"] Nov 01 10:25:15.091439 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:15.091426 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk"] Nov 01 10:25:15.092840 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:15.092821 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-btkkk"] Nov 01 10:25:15.477403 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:25:15.477359 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:25:17.057205 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:17.057167 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604f74a1-fe23-48cd-b7b5-3151fda3ce28" path="/var/lib/kubelet/pods/604f74a1-fe23-48cd-b7b5-3151fda3ce28/volumes" Nov 01 10:25:17.057761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:17.057742 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c63098-43de-4ccc-a9d7-57623b19a384" path="/var/lib/kubelet/pods/d4c63098-43de-4ccc-a9d7-57623b19a384/volumes" Nov 01 10:25:17.598134 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:17.598099 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:25:45.477290 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:25:45.477198 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:25:54.377117 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:25:54.377074 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:26:01.546563 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:01.546520 5710 scope.go:122] "RemoveContainer" containerID="35c9ff9a73c4a7e85f6326af97a0821acedbe18e8ac642b8f929661988058511" Nov 01 10:26:01.562143 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:01.562111 5710 scope.go:122] "RemoveContainer" containerID="99fb4d58df9d508f5fda3fc42cb91844f5d996c58c3d9f2d40841cce3a4786de" Nov 01 10:26:04.895948 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:04.895901 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:26:04.896482 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:04.895994 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:26:07.185984 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:07.185952 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:26:15.477779 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:26:15.477736 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:26:31.171199 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:31.171162 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:26:43.961990 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:26:43.961955 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:26:45.490989 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:26:45.490942 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:27:01.601703 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:01.601672 5710 scope.go:122] "RemoveContainer" containerID="0cdc8bd554eb33b620135f5bb66accdabc4e0f1508eaf8f2a70e180e3eb19e1f" Nov 01 10:27:01.613509 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:01.613487 5710 scope.go:122] "RemoveContainer" containerID="5e9632b94996120e0d3a96aa18ea5bd82ba6b0fc8fc95a84396a0233d4aaac4e" Nov 01 10:27:15.477143 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:27:15.477093 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:27:15.799966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:15.799889 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:27:26.458107 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:26.458064 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:27:34.901756 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:34.900894 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:27:34.901756 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:34.901057 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:27:37.786901 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:37.786864 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:27:44.050696 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:44.050659 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh"] Nov 01 10:27:44.102966 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:44.102929 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xpdxh"] Nov 01 10:27:45.056902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:45.056870 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15f2861-719f-401e-a6d4-6e23ea760b8e" path="/var/lib/kubelet/pods/f15f2861-719f-401e-a6d4-6e23ea760b8e/volumes" Nov 01 10:27:45.477824 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:27:45.477788 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:27:49.085065 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:27:49.085021 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:28:01.596432 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:01.596273 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:28:01.605149 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:01.605124 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jvnzm_58067792-becb-4036-a80e-be1522d0f974/ovn-acl-logging/0.log" Nov 01 10:28:12.568829 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:12.568792 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:28:15.480461 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:28:15.480405 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:28:23.257659 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:23.257612 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:28:31.575292 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:31.575240 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:28:45.477496 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:28:45.477415 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:28:49.898743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:49.897861 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:28:49.898743 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:49.897983 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:28:52.456061 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:28:52.456026 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:29:01.646262 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:01.646229 5710 scope.go:122] "RemoveContainer" containerID="bfaab4c85196cd4b7119fa355193b2ef8547bc9774e2c2226d6c2f3544ae462b" Nov 01 10:29:15.477014 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:29:15.476971 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:29:16.480620 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:16.480579 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:29:19.898746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:19.897946 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:29:19.898746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:19.898017 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:29:36.050708 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:36.050667 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz"] Nov 01 10:29:36.102396 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:36.102361 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-97xxz"] Nov 01 10:29:37.060486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:37.060441 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43582e8-aeaf-4bbe-b964-3255a9e2b38b" path="/var/lib/kubelet/pods/e43582e8-aeaf-4bbe-b964-3255a9e2b38b/volumes" Nov 01 10:29:39.353309 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:39.352848 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hkrgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:29:39.353309 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:39.352945 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hkrgg" podUID="2aed84f7-2b75-4239-b5f3-cbc504023f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:29:45.477627 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:29:45.477582 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:29:56.050641 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:56.050604 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7"] Nov 01 10:29:56.102647 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:56.102608 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vl6s7"] Nov 01 10:29:57.058761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:29:57.058712 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d827009-d262-40f7-93c0-0b1c68039f40" path="/var/lib/kubelet/pods/6d827009-d262-40f7-93c0-0b1c68039f40/volumes" Nov 01 10:30:00.165279 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.165233 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn"] Nov 01 10:30:00.166099 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.166064 5710 cpu_manager.go:414] "containerMap: removing container" podUID="d4c63098-43de-4ccc-a9d7-57623b19a384" containerName="maas-api" Nov 01 10:30:00.166296 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.166109 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c63098-43de-4ccc-a9d7-57623b19a384" containerName="maas-api" Nov 01 10:30:00.166296 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.166249 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c63098-43de-4ccc-a9d7-57623b19a384" containerName="maas-api" Nov 01 10:30:00.170012 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.169990 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.228470 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.228433 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn"] Nov 01 10:30:00.279153 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.279115 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2f29k\"" Nov 01 10:30:00.361481 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.361438 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892150-x6wrn\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.361891 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.361864 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgwf\" (UniqueName: \"kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf\") pod \"maas-api-key-cleanup-29892150-x6wrn\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.465201 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.465097 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892150-x6wrn\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.465504 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.465478 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgwf\" (UniqueName: \"kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf\") pod \"maas-api-key-cleanup-29892150-x6wrn\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.466597 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.466568 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892150-x6wrn\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.544795 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.537978 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgwf\" (UniqueName: \"kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf\") pod \"maas-api-key-cleanup-29892150-x6wrn\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:00.803803 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:00.795906 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:01.318748 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:01.315704 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn"] Nov 01 10:30:01.345448 ip-10-0-143-16 kubenswrapper[5710]: W1101 10:30:01.345397 5710 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770705d2_695b_42dc_9ecd_602292c50583.slice/crio-0466305e1b690145e11c6c8d55e701129c6767e0c5706c7e010884235819ac69 WatchSource:0}: Error finding container 0466305e1b690145e11c6c8d55e701129c6767e0c5706c7e010884235819ac69: Status 404 returned error can't find the container with id 0466305e1b690145e11c6c8d55e701129c6767e0c5706c7e010884235819ac69 Nov 01 10:30:01.773791 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:01.773749 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" event={"ID":"770705d2-695b-42dc-9ecd-602292c50583","Type":"ContainerStarted","Data":"0466305e1b690145e11c6c8d55e701129c6767e0c5706c7e010884235819ac69"} Nov 01 10:30:02.781346 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:02.781310 5710 generic.go:356] "Generic (PLEG): container finished" podID="770705d2-695b-42dc-9ecd-602292c50583" containerID="7e938fa051783876aa2e09dbb1decdda75c206e7266f86fd055bbf7dd2501c4e" exitCode=60 Nov 01 10:30:02.781837 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:02.781397 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" event={"ID":"770705d2-695b-42dc-9ecd-602292c50583","Type":"ContainerDied","Data":"7e938fa051783876aa2e09dbb1decdda75c206e7266f86fd055bbf7dd2501c4e"} Nov 01 10:30:02.781837 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:02.781640 5710 scope.go:122] "RemoveContainer" containerID="7e938fa051783876aa2e09dbb1decdda75c206e7266f86fd055bbf7dd2501c4e" Nov 01 10:30:03.788415 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:03.788363 5710 generic.go:356] "Generic (PLEG): container finished" podID="770705d2-695b-42dc-9ecd-602292c50583" containerID="8cd6d5e0e5e6475da7a86e9b0e61d717257a110c13ab601d1928daa671560d30" exitCode=60 Nov 01 10:30:03.788920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:03.788452 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" event={"ID":"770705d2-695b-42dc-9ecd-602292c50583","Type":"ContainerDied","Data":"8cd6d5e0e5e6475da7a86e9b0e61d717257a110c13ab601d1928daa671560d30"} Nov 01 10:30:03.788920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:03.788531 5710 scope.go:122] "RemoveContainer" containerID="7e938fa051783876aa2e09dbb1decdda75c206e7266f86fd055bbf7dd2501c4e" Nov 01 10:30:03.788920 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:03.788846 5710 scope.go:122] "RemoveContainer" containerID="8cd6d5e0e5e6475da7a86e9b0e61d717257a110c13ab601d1928daa671560d30" Nov 01 10:30:03.789130 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:30:03.789109 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892150-x6wrn_odh-ai-gateway-infra(770705d2-695b-42dc-9ecd-602292c50583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" podUID="770705d2-695b-42dc-9ecd-602292c50583" Nov 01 10:30:04.794841 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:04.794770 5710 scope.go:122] "RemoveContainer" containerID="8cd6d5e0e5e6475da7a86e9b0e61d717257a110c13ab601d1928daa671560d30" Nov 01 10:30:04.795282 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:30:04.795031 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892150-x6wrn_odh-ai-gateway-infra(770705d2-695b-42dc-9ecd-602292c50583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" podUID="770705d2-695b-42dc-9ecd-602292c50583" Nov 01 10:30:12.664901 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:12.664842 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-gsql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:30:12.665467 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:12.664919 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-gsql5" podUID="eec42fe9-a657-4b42-bb54-7849913616b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:30:15.476187 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:30:15.476148 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:30:17.058891 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:17.058859 5710 scope.go:122] "RemoveContainer" containerID="8cd6d5e0e5e6475da7a86e9b0e61d717257a110c13ab601d1928daa671560d30" Nov 01 10:30:17.885058 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:17.885021 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" event={"ID":"770705d2-695b-42dc-9ecd-602292c50583","Type":"ContainerStarted","Data":"be171b5b37ab68d24d127c61706a4a08fc6352300bf2ed389c1444d4e3b77032"} Nov 01 10:30:18.010060 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:18.010003 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" podStartSLOduration=18.009989502 podStartE2EDuration="18.009989502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:30:18.009090975 +0000 UTC m=+7041.097427659" watchObservedRunningTime="2026-11-01 10:30:18.009989502 +0000 UTC m=+7041.098326172" Nov 01 10:30:18.891880 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:18.891845 5710 generic.go:356] "Generic (PLEG): container finished" podID="770705d2-695b-42dc-9ecd-602292c50583" containerID="be171b5b37ab68d24d127c61706a4a08fc6352300bf2ed389c1444d4e3b77032" exitCode=60 Nov 01 10:30:18.892383 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:18.891922 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" event={"ID":"770705d2-695b-42dc-9ecd-602292c50583","Type":"ContainerDied","Data":"be171b5b37ab68d24d127c61706a4a08fc6352300bf2ed389c1444d4e3b77032"} Nov 01 10:30:18.892383 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:18.891999 5710 scope.go:122] "RemoveContainer" containerID="8cd6d5e0e5e6475da7a86e9b0e61d717257a110c13ab601d1928daa671560d30" Nov 01 10:30:18.892510 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:18.892402 5710 scope.go:122] "RemoveContainer" containerID="be171b5b37ab68d24d127c61706a4a08fc6352300bf2ed389c1444d4e3b77032" Nov 01 10:30:18.892711 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:30:18.892683 5710 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892150-x6wrn_odh-ai-gateway-infra(770705d2-695b-42dc-9ecd-602292c50583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" podUID="770705d2-695b-42dc-9ecd-602292c50583" Nov 01 10:30:19.017472 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:19.017441 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn"] Nov 01 10:30:20.031015 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.030984 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:20.130422 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.130381 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf\" (UniqueName: \"kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf\") pod \"770705d2-695b-42dc-9ecd-602292c50583\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " Nov 01 10:30:20.130656 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.130451 5710 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca\") pod \"770705d2-695b-42dc-9ecd-602292c50583\" (UID: \"770705d2-695b-42dc-9ecd-602292c50583\") " Nov 01 10:30:20.130895 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.130868 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca" pod "770705d2-695b-42dc-9ecd-602292c50583" (UID: "770705d2-695b-42dc-9ecd-602292c50583"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 10:30:20.132865 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.132843 5710 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf" pod "770705d2-695b-42dc-9ecd-602292c50583" (UID: "770705d2-695b-42dc-9ecd-602292c50583"). InnerVolumeSpecName "kube-api-access-hdgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 10:30:20.231902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.231806 5710 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgwf\" (UniqueName: \"kubernetes.io/projected/770705d2-695b-42dc-9ecd-602292c50583-kube-api-access-hdgwf\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:30:20.231902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.231840 5710 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/770705d2-695b-42dc-9ecd-602292c50583-maas-api-service-ca\") on node \"ip-10-0-143-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 10:30:20.906784 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.906739 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" event={"ID":"770705d2-695b-42dc-9ecd-602292c50583","Type":"ContainerDied","Data":"0466305e1b690145e11c6c8d55e701129c6767e0c5706c7e010884235819ac69"} Nov 01 10:30:20.906972 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.906826 5710 scope.go:122] "RemoveContainer" containerID="be171b5b37ab68d24d127c61706a4a08fc6352300bf2ed389c1444d4e3b77032" Nov 01 10:30:20.906972 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:20.906764 5710 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn" Nov 01 10:30:21.078365 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:21.078326 5710 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn"] Nov 01 10:30:21.085576 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:21.085533 5710 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892150-x6wrn"] Nov 01 10:30:23.057665 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:23.057632 5710 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770705d2-695b-42dc-9ecd-602292c50583" path="/var/lib/kubelet/pods/770705d2-695b-42dc-9ecd-602292c50583/volumes" Nov 01 10:30:34.901746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:34.899855 5710 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vmqkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 10:30:34.901746 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:30:34.899937 5710 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vmqkd" podUID="98262e95-96f5-4250-ac07-edfdb9334102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 10:30:45.480234 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:30:45.480194 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:31:01.678913 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:01.678877 5710 scope.go:122] "RemoveContainer" containerID="cbfc04a4ef1bd08c2edc0d1a3d8c1b2dcdea1851963b051c4a622b4d6d7cea3d" Nov 01 10:31:01.687962 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:01.687938 5710 scope.go:122] "RemoveContainer" containerID="cfbf02c3dfcc27a68e17c1d51eb64bf59bc00bd86ef13f86208e6f2330e3ac1a" Nov 01 10:31:14.135460 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:14.135426 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2rfzc"] Nov 01 10:31:15.493860 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:31:15.493512 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:31:28.777865 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:28.777821 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c65545f6c-spmfl_1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa/authorino/0.log" Nov 01 10:31:40.423822 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:40.423734 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7657794547-t5tw5_ab7ffc9a-2189-49be-adcb-2ff757eb57d5/manager/0.log" Nov 01 10:31:42.315766 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:42.315707 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5w7w2_85247c14-ee89-4497-9d63-7518733b0a98/manager/0.log" Nov 01 10:31:43.543619 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:43.543586 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-676b7bff79-6h7t6_c0558595-c99f-4be6-8ff8-5d9cfc349a9e/maas-api/0.log" Nov 01 10:31:45.476782 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:31:45.476744 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:31:49.496209 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:49.496171 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c65545f6c-spmfl_1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa/authorino/0.log" Nov 01 10:31:50.973547 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:50.973511 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2rfzc_308c91a4-a197-4420-8d55-7d7e9fe1e44d/limitador/0.log" Nov 01 10:31:52.639131 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:52.639097 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t8wbh_d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6/discovery/0.log" Nov 01 10:31:52.901712 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:52.901620 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649b5c55d-f88db_f1e8428b-1e87-425d-91e9-02dcc567fcb6/kube-auth-proxy/0.log" Nov 01 10:31:54.301684 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:54.301648 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-gsql5_eec42fe9-a657-4b42-bb54-7849913616b4/istio-proxy/0.log" Nov 01 10:31:54.777198 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:54.777162 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-hkrgg_2aed84f7-2b75-4239-b5f3-cbc504023f8e/istio-proxy/0.log" Nov 01 10:31:55.452854 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:55.452822 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-vmqkd_98262e95-96f5-4250-ac07-edfdb9334102/istio-proxy/0.log" Nov 01 10:31:55.899135 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:55.899101 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-549c5f7d67-lwtfl_8d3ec379-0842-4328-95fe-f2b1f007aabb/payload-pre-processing/0.log" Nov 01 10:31:57.374940 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:57.374909 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xvmvg_81739fa3-884f-402f-a010-2c5c8f2e4943/main/0.log" Nov 01 10:31:57.642740 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:57.642615 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ct5sc_8a5af8c7-0109-4d77-ab61-837d1b8f79e9/main/0.log" Nov 01 10:31:57.869341 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:57.869305 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lnhsp_0c2579b5-5533-45d8-a0b6-c4272b055f20/main/0.log" Nov 01 10:31:58.321544 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:58.321509 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qqzhj_d1298d16-27bd-4638-8969-39eef9ffd697/main/0.log" Nov 01 10:31:58.768625 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:31:58.768589 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7q7k8_814d41ce-7e8f-4d90-a1e4-8cde4feb47ce/main/0.log" Nov 01 10:32:15.479907 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:32:15.479868 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:32:19.022913 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:19.022875 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tg7cb_fc68d9cc-01a2-49b3-b0f6-f510d04caefa/global-pull-secret-syncer/0.log" Nov 01 10:32:20.358486 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:20.358449 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7657794547-t5tw5_ab7ffc9a-2189-49be-adcb-2ff757eb57d5/manager/0.log" Nov 01 10:32:22.090072 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:22.090018 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t58md_958489ac-f598-4365-a166-22371ca307d4/konnectivity-agent/0.log" Nov 01 10:32:23.330819 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:23.330779 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-16.us-west-2.compute.internal_61735862d7ea6a76be3182acee32ca86/haproxy/0.log" Nov 01 10:32:24.372597 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:24.372558 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5w7w2_85247c14-ee89-4497-9d63-7518733b0a98/manager/0.log" Nov 01 10:32:27.210497 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:27.210456 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c65545f6c-spmfl_1b941aa0-8dbc-4dec-a4fa-2f8d5dafc3aa/authorino/0.log" Nov 01 10:32:27.297687 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:27.297651 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7657794547-t5tw5_ab7ffc9a-2189-49be-adcb-2ff757eb57d5/manager/0.log" Nov 01 10:32:28.798159 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798121 5710 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b"] Nov 01 10:32:28.798698 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798679 5710 cpu_manager.go:414] "containerMap: removing container" podUID="770705d2-695b-42dc-9ecd-602292c50583" containerName="cleanup" Nov 01 10:32:28.798698 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798699 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770705d2-695b-42dc-9ecd-602292c50583" containerName="cleanup" Nov 01 10:32:28.798830 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798754 5710 cpu_manager.go:414] "containerMap: removing container" podUID="770705d2-695b-42dc-9ecd-602292c50583" containerName="cleanup" Nov 01 10:32:28.798830 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798762 5710 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770705d2-695b-42dc-9ecd-602292c50583" containerName="cleanup" Nov 01 10:32:28.798830 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798826 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="770705d2-695b-42dc-9ecd-602292c50583" containerName="cleanup" Nov 01 10:32:28.798930 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.798840 5710 memory_manager.go:365] "RemoveStaleState removing state" podUID="770705d2-695b-42dc-9ecd-602292c50583" containerName="cleanup" Nov 01 10:32:28.801992 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.801976 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.834612 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.834584 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-proc\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.834793 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.834661 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-sys\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.834880 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.834835 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-lib-modules\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.834946 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.834891 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pstj\" (UniqueName: \"kubernetes.io/projected/fee87d35-6da9-4106-8505-1d6a33fb07ca-kube-api-access-6pstj\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.835004 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.834974 5710 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-podres\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.908893 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.908846 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w96kq\"/\"kube-root-ca.crt\"" Nov 01 10:32:28.908893 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.908882 5710 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w96kq\"/\"default-dockercfg-f9ll4\"" Nov 01 10:32:28.909123 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.908923 5710 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w96kq\"/\"openshift-service-ca.crt\"" Nov 01 10:32:28.910489 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.910459 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b"] Nov 01 10:32:28.935675 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935640 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-sys\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.935902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935694 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-lib-modules\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.935902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935768 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pstj\" (UniqueName: \"kubernetes.io/projected/fee87d35-6da9-4106-8505-1d6a33fb07ca-kube-api-access-6pstj\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.935902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935794 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-podres\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.935902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935797 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-sys\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.935902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935878 5710 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-proc\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.935902 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935894 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-podres\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.936200 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935899 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-lib-modules\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.936200 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.935955 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fee87d35-6da9-4106-8505-1d6a33fb07ca-proc\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:28.999431 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:28.999397 5710 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pstj\" (UniqueName: \"kubernetes.io/projected/fee87d35-6da9-4106-8505-1d6a33fb07ca-kube-api-access-6pstj\") pod \"perf-node-gather-daemonset-97l9b\" (UID: \"fee87d35-6da9-4106-8505-1d6a33fb07ca\") " pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:29.115239 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:29.115197 5710 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:29.508780 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:29.508632 5710 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b"] Nov 01 10:32:29.866107 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:29.866067 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" event={"ID":"fee87d35-6da9-4106-8505-1d6a33fb07ca","Type":"ContainerStarted","Data":"4422dde4b4bcfc601011f6b0fd48ba41c7ae9df03005b8b24ba533f3029f0f8a"} Nov 01 10:32:29.866761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:29.866128 5710 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" event={"ID":"fee87d35-6da9-4106-8505-1d6a33fb07ca","Type":"ContainerStarted","Data":"22add80fb3ebbed432159fd26c492c78818b15f7d3b33678dcb7cb849a8f52f5"} Nov 01 10:32:29.866761 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:29.866187 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:29.987025 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:29.986950 5710 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" podStartSLOduration=1.986932041 podStartE2EDuration="1.986932041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 10:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 10:32:29.985942199 +0000 UTC m=+7173.074278870" watchObservedRunningTime="2026-11-01 10:32:29.986932041 +0000 UTC m=+7173.075268739" Nov 01 10:32:30.501187 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:30.501156 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2rfzc_308c91a4-a197-4420-8d55-7d7e9fe1e44d/limitador/0.log" Nov 01 10:32:31.323659 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:31.323608 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5w7w2_85247c14-ee89-4497-9d63-7518733b0a98/manager/0.log" Nov 01 10:32:33.597103 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:33.597074 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/alertmanager/0.log" Nov 01 10:32:34.020683 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:34.020592 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/config-reloader/0.log" Nov 01 10:32:34.206922 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:34.206889 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7657794547-t5tw5_ab7ffc9a-2189-49be-adcb-2ff757eb57d5/manager/0.log" Nov 01 10:32:34.442402 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:34.442370 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/kube-rbac-proxy-web/0.log" Nov 01 10:32:34.863063 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:34.863032 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/kube-rbac-proxy/0.log" Nov 01 10:32:35.286939 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:35.286857 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/kube-rbac-proxy-metric/0.log" Nov 01 10:32:35.707266 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:35.707220 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/prom-label-proxy/0.log" Nov 01 10:32:35.884204 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:35.884176 5710 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w96kq/perf-node-gather-daemonset-97l9b" Nov 01 10:32:36.141394 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:36.141361 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21a6f7a9-e5f9-4a81-9808-007cac6ccac0/init-config-reloader/0.log" Nov 01 10:32:37.025380 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:37.025349 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vv4sc_d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/kube-state-metrics/0.log" Nov 01 10:32:37.437346 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:37.437315 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vv4sc_d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/kube-rbac-proxy-main/0.log" Nov 01 10:32:37.901051 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:37.901015 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vv4sc_d398bcb3-f7fc-4307-bbc3-6acccfcdc43c/kube-rbac-proxy-self/0.log" Nov 01 10:32:38.027059 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:38.027023 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5w7w2_85247c14-ee89-4497-9d63-7518733b0a98/manager/0.log" Nov 01 10:32:38.757648 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:38.757614 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hcmzn_0e698209-ef38-47d5-be80-4e5208813720/monitoring-plugin/0.log" Nov 01 10:32:40.520789 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:40.520760 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p7p9g_7a28283c-3f57-4276-a7d1-a7792377f37b/node-exporter/0.log" Nov 01 10:32:40.947180 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:40.947145 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p7p9g_7a28283c-3f57-4276-a7d1-a7792377f37b/kube-rbac-proxy/0.log" Nov 01 10:32:41.366389 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:41.366354 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p7p9g_7a28283c-3f57-4276-a7d1-a7792377f37b/init-textfile/0.log" Nov 01 10:32:43.068159 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:43.068120 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pd9k5_ee31b8bb-4ebe-41dd-a701-66eaf42137da/kube-rbac-proxy-main/0.log" Nov 01 10:32:43.491286 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:43.491237 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pd9k5_ee31b8bb-4ebe-41dd-a701-66eaf42137da/kube-rbac-proxy-self/0.log" Nov 01 10:32:43.920456 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:43.920416 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pd9k5_ee31b8bb-4ebe-41dd-a701-66eaf42137da/openshift-state-metrics/0.log" Nov 01 10:32:45.489162 ip-10-0-143-16 kubenswrapper[5710]: E1101 10:32:45.489115 5710 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bm9bj" containerName="extractor" Nov 01 10:32:45.695054 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:45.695018 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t8wbh_d0a861ef-ebbf-4b75-a0ab-58cd36f8b8a6/discovery/0.log" Nov 01 10:32:45.800147 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:45.800041 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649b5c55d-f88db_f1e8428b-1e87-425d-91e9-02dcc567fcb6/kube-auth-proxy/0.log" Nov 01 10:32:47.059396 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:47.059368 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-gsql5_eec42fe9-a657-4b42-bb54-7849913616b4/istio-proxy/0.log" Nov 01 10:32:47.336675 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:47.336599 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vssf9_22098de3-a796-47bb-98c9-30991caa95ac/prometheus-operator/0.log" Nov 01 10:32:47.521788 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:47.521710 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-hkrgg_2aed84f7-2b75-4239-b5f3-cbc504023f8e/istio-proxy/0.log" Nov 01 10:32:47.779317 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:47.779263 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vssf9_22098de3-a796-47bb-98c9-30991caa95ac/kube-rbac-proxy/0.log" Nov 01 10:32:48.152500 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:48.152467 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-vmqkd_98262e95-96f5-4250-ac07-edfdb9334102/istio-proxy/0.log" Nov 01 10:32:48.203033 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:48.203004 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2j2r6_d0c527af-be39-4887-990a-0dd793e2a751/prometheus-operator-admission-webhook/0.log" Nov 01 10:32:48.563401 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:48.563318 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-549c5f7d67-lwtfl_8d3ec379-0842-4328-95fe-f2b1f007aabb/payload-pre-processing/0.log" Nov 01 10:32:48.630463 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:48.630417 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bf5b6787f-ckmvn_dfbd8ec8-8e1a-436c-b5cc-493861c60fe1/telemeter-client/0.log" Nov 01 10:32:49.051046 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:49.050998 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bf5b6787f-ckmvn_dfbd8ec8-8e1a-436c-b5cc-493861c60fe1/reload/0.log" Nov 01 10:32:49.472334 ip-10-0-143-16 kubenswrapper[5710]: I1101 10:32:49.472301 5710 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bf5b6787f-ckmvn_dfbd8ec8-8e1a-436c-b5cc-493861c60fe1/kube-rbac-proxy/0.log"